Order queue open · Verified lists delivered within 24–48 hours
How to run A/B tests in Instantly 2026: how to set up variants, what sample sizes matter, what to test first, how to read results, and how to apply findings to future campaigns.
Sarah Okonkwo
Sales ops specialist, deliverability obsessive · Updated June 24, 2026
Last updated: June 2026 · Sarah Okonkwo, Sales ops specialist, deliverability obsessive
TL;DR — 5 things to know before reading
A/B testing is the systematic way to improve cold email performance over time, but it is frequently misused in two ways: testing too many variables at once (which makes results uninterpretable) and running tests on samples too small to be statistically meaningful (which produces confident-looking results that are actually noise).
Used correctly, A/B testing in Instantly is the mechanism by which a 9% reply rate becomes 14% over 8 weeks of iteration. Each test produces one learnable insight; that insight updates one element of future campaigns; the next campaign starts from the improved baseline. Quarvio provides the verified contact volume needed to run tests with statistically meaningful sample sizes; Inframail provides the warmed inboxes that deliver tests to the inbox rather than spam; and Aimfox covers the LinkedIn parallel channel where a separate set of messaging tests runs independently.
This guide covers the full A/B testing system: what to test in what order, how to set up each test correctly in Instantly, how to calculate whether a result is meaningfully different from noise, how to read and apply results, and the advanced testing tactics that compound improvements over time. Every section includes failure modes — the specific mistakes that produce either wasted tests or false confidence in bad data.
Most practitioners who start A/B testing expect linear improvement: one good test, one improvement. The compounding effect is what most people miss.
When you test and validate the subject line, every future campaign uses the validated subject line. When you then test and validate the opening sentence on top of the improved subject line, the starting point is already stronger. When you then test and validate the CTA on top of both improved elements, the effect compounds. A campaign where you have tested and validated the subject line, opening sentence, and CTA separately is likely to perform 30–50% better than a campaign where none of those elements have been tested — not because any single test produced a 30% improvement, but because each test produced a 10–15% improvement that stacked on the previous one.
Per Instantly's cold email benchmark report, the average reply rate across cold email campaigns is 3.43%. Top-quartile practitioners consistently reach 10% and above. That gap is not explained by better lists or better tools in isolation — it is explained by systematic iteration over time. Testing is the mechanism.
Not all test variables produce equal learning. Test in this priority order:
Subject lines are the gating variable for open rate. A test between two subject line approaches produces the clearest, most actionable learnings because open rate is directly attributable to what happened before the email was opened — meaning the subject line (and sender name) are the only variables that matter.
What to test between subject line variants:
Format: Question format ("Quick question about [Company]'s outbound") vs. statement format ("How [Company type] teams cut prospecting time by 40%"). Question format typically produces higher open rates for curiosity-driven ICPs; statement format works better for outcome-focused ICPs.
Length: Short (under 5 words) vs. medium (6–9 words) vs. long (10+ words). There is no universal winner — ICP matters more than length. Finance and legal ICPs tend to open longer, more specific subjects. Startup and tech ICPs often open short, direct subjects faster.
Personalization token: First name vs. company name vs. industry reference vs. no personalization. Testing personalization depth directly in the subject line often produces surprising results.
Number inclusion: "3 ways [ICP] reduce [pain]" vs. the non-numbered equivalent. Number inclusion typically improves open rates because readers understand exactly what they are committing to when they open.
Bracket formatting: "[Quick question]" or "[Case study]" as a prefix before the actual subject vs. no bracket. Some ICPs respond well to this format signal; others find it sales-obvious.
The first sentence of the email determines whether the recipient reads to the end. Since almost all reply decisions are made in the first 30 seconds of reading, the opening sentence is the highest-leverage element for improving reply rate.
Opening sentence structures to test against each other:
Insight opener: "Most [ICP] teams we speak with are dealing with [specific operational problem]." Demonstrates research without being intrusive; works well for analytical ICPs.
Outcome opener: "We helped [comparable company type] cut [metric] by [number] in [timeframe]." Leads with proof; works well when you have strong social proof and a results-oriented ICP.
Pain opener: "If you're seeing [specific problem], it's likely because [root cause]." Demonstrates domain knowledge; works well when the pain point is highly specific and validated.
Peer reference opener: "[Number] of [ICP in specific market] use [solution] to [outcome]." Creates social proof without naming specific clients; works well for competitive, FOMO-driven ICPs.
Some audiences respond to concise emails (80–120 words), others engage with more detailed context (200–300 words). Test short vs. long to find the preference for your specific ICP. The benchmark: Woodpecker's 2025 cold email benchmark study notes that shorter emails (under 150 words) consistently outperform longer ones for cold outreach, but this is an average across ICPs. Some high-consideration buyers (executive buyers, technical buyers) engage better with slightly longer, more substantive first-touch emails.
"Worth a 15-minute call?" vs. "Would you be open to a quick call?" vs. "Happy to share a case study — interested?" Each asks for different commitment levels and produces different response rates. Low-commitment CTAs ("Worth a quick look?") typically produce higher total reply rates but more low-intent responses. High-commitment CTAs ("Ready to see a demo?") produce lower total reply rates but higher intent responses. Test which end of the spectrum your ICP falls on.
Highly specific AI-generated personalised openers vs. role-based personalisation vs. company-based personalisation — test which level of personalisation your ICP responds to. The common assumption that more personalisation always wins is frequently wrong in practice. Role-based personalisation ("[Title] teams at [company type]") often outperforms heavy AI personalisation in efficiency-to-result terms, especially at scale.
The number of days between sequence steps affects both reply rate and unsubscribe rate. A 3-day gap vs. a 5-day gap between step 1 and step 2 can produce meaningful differences in reply volume. Test this last because the signal is difficult to isolate from other variables and the test period is longer.
This section covers what most cold email A/B testing guides skip: whether a result is genuinely meaningful or just noise.
Statistical significance answers the question: "If I ran this test again with a fresh set of contacts, how likely is it that the winner would win again?" At 95% confidence, you are saying: "I am 95% confident this is a real difference, not a random fluctuation."
In practice, 95% confidence in cold email requires sample sizes that most campaigns cannot easily achieve. The minimum detectable effect at 95% confidence for a change from 8% to 10% reply rate (a 2 percentage point, 25% relative improvement) requires approximately 1,000 sends per variant. Few campaigns have this volume available for a single test.
For cold email, a more practical standard is useful confidence — defined as: "Am I confident enough in this result to update my template based on it?" The practical rules:
For open rate tests:
For reply rate tests:
A difference that meets these criteria is worth acting on. A difference below these criteria requires re-running the test with a larger sample before changing your template.
In e-commerce or SaaS product A/B testing, you typically have thousands of daily sessions and can reach statistical confidence in 7 days. Cold email has two features that make this harder:
The solution is a longer test window and larger samples — not more sophisticated statistics.
Before opening Instantly, write the hypothesis in this format: "If we change [specific element] from [current version] to [new version], we expect [specific metric] to improve by at least [threshold], because [reason based on ICP knowledge]."
Sub-step 1.1: State the test in if-then format. "If we change the subject line from a statement format to a question format, we expect open rate to increase by at least 5 percentage points, because our VP of Sales ICP tends to engage with curiosity-based openers based on our prior reply content analysis."
Sub-step 1.2: Define the primary success metric before starting. Open rate for subject line tests. Reply rate for body copy tests. Positive reply rate for CTA tests.
Sub-step 1.3: Confirm one variable only. Review both variant drafts and confirm the only difference is the one element under test. If you spot additional differences, standardize them before running the test.
Sub-step 1.4: Set the minimum acceptance threshold. "The winner needs a 5+ percentage point open rate advantage at 200+ sends per variant to update the template."
Benchmark: A well-formed hypothesis takes 10 minutes to write and makes the result interpretation unambiguous when the test completes.
Failure mode: Vague hypothesis ("Let's see which subject line performs better") means you will interpret any result as confirming your prior beliefs. A 0.5% open rate difference will be called a "win" because there is no pre-stated threshold.
Sub-step 2.1: Count the available contacts for this ICP. A/B tests require contacts that have not been emailed in this campaign before. If your ICP pool has 500 contacts, you have enough for a subject line test (200 per variant) with 100 left over.
Sub-step 2.2: Calculate the required send volume and timeline. If you send 50 contacts per day total, a 400-contact test (200 per variant) completes sends in 8 days. Add 6 days for lagged responses = 14-day evaluation window.
Sub-step 2.3: Verify contact recency. If contacts were sourced more than 12 months ago, re-verify before running the test. High bounce rates will contaminate the test results. Quarvio delivers pre-verified contacts specifically to avoid this problem.
Sub-step 2.4: Remove contacts already in the sequence or who have received prior outreach from this campaign. Duplicate touches contaminate test results.
Benchmark: A clean audience segment for A/B testing is: 400+ contacts, sourced within 12 months, verified, and not previously messaged in this campaign.
Failure mode: Running an A/B test on 180 contacts total (90 per variant) because "the ICP pool is small." Results from 90 sends per variant are noise. Either find more contacts or skip the A/B test and use the full pool for a single validated sequence.
Sub-step 3.1: Navigate to Campaigns → New Campaign. Create the base campaign with Variant A (your current or control version).
Sub-step 3.2: In the sequence editor, add Variant B to the specific step you are testing. Modify only the one element under test. If testing subject lines: change only the subject line field in Variant B. Leave the email body, CTA, and all other settings identical.
Sub-step 3.3: Cross-check variants for accidental differences. Read Variant A and Variant B aloud. Any difference you did not intend to test must be corrected before launching.
Sub-step 3.4: Configure sending inboxes. Both variants should use the same inbox rotation settings. If Variant A sends from Inbox 1 and Inbox 2, Variant B should also send from Inbox 1 and Inbox 2. Different inbox assignments introduce a confounding variable (inbox reputation) into the test.
Benchmark: The only difference between Variant A and Variant B after Step 3 should be the single element you identified in your hypothesis.
Failure mode: Accidentally leaving a different call-to-action phrasing in Variant B while also testing the subject line. The test is now testing two things simultaneously and results cannot be attributed to either change.
Sub-step 4.1: In the campaign sequence editor, find the variant split settings.
Sub-step 4.2: Set the split percentage:
Sub-step 4.3: Confirm Instantly is distributing contacts randomly between variants, not by time of day or send order. Time-based distribution (all Monday sends go to Variant A, Tuesday sends to Variant B) introduces a weekday-effect confound.
Sub-step 4.4: Record the split ratio in your test log.
Benchmark: 50/50 split is correct for more than 90% of A/B tests. The only reason to use 80/20 is protecting an established baseline performing above 12% reply rate.
Failure mode: Setting a 90/10 split because you "have a feeling" Variant B is better and don't want to risk too much campaign performance. A 90/10 split means Variant B will not reach minimum sample size without running the test far longer than the 14-day window.
Sub-step 5.1: Using the send volume calculation from Step 2, confirm that the campaign will reach minimum sample size within the 14-day test window. If not, either increase daily send volume or extend the test window accordingly.
Sub-step 5.2: Create a test log entry. Record: test start date, campaign name, ICP description, hypothesis, Variant A (control), Variant B (test), primary success metric, acceptance threshold, expected completion date. One row per test.
Sub-step 5.3: Confirm that Instantly will not auto-optimize (auto-pause the losing variant) before minimum sample is reached. If Instantly's AI optimization feature is enabled, it may intervene before you have statistically useful data. For controlled tests, disable auto-optimization for the test duration.
Sub-step 5.4: Set a calendar reminder for the evaluation date (start date + 14 days).
Benchmark: Minimum sample sizes by test type:
| Test type | Minimum sends per variant |
|---|---|
| Subject line (open rate) | 200+ |
| Opening sentence (reply rate) | 300+ |
| Email length (reply rate) | 300+ |
| CTA phrasing (reply rate) | 400+ |
| Personalization depth (reply rate) | 400+ |
Failure mode: Sending 10 contacts per day and expecting meaningful results in 14 days (140 total sends, 70 per variant). This is well below minimum sample for any metric. Increase daily volume or wait until you have more contacts available.
Sub-step 6.1: Check metrics after the first 24 hours only for critical failures: bounce rate above 10% (requires immediate pause), spam complaint spike (requires immediate pause). Do not check open or reply rate results.
Sub-step 6.2: Record daily send counts (not results) in the test log. Track: date, sends to Variant A, sends to Variant B, running totals. This confirms the split is working as configured.
Sub-step 6.3: Do not check variant performance before reaching 80% of minimum sample size. Looking at early results creates confirmation bias: whichever variant is ahead after 5 days will look like the obvious winner regardless of whether the difference is real.
Sub-step 6.4: Note any external events during the test period that could affect results. A major industry news event, a public holiday, or end-of-quarter effects on your ICP can cause temporary spikes or drops in reply rate. Note these events in the test log so they can be factored into result interpretation.
Benchmark: The only valid reason to intervene before minimum sample is a critical metric (bounce rate above 10%, spam complaint above 0.3%).
Failure mode: Pausing the losing variant after day 3 because it is "obviously" underperforming. Early results are high-variance. The variant that leads after day 3 with 60 sends each is not a validated winner — the lead may completely reverse by day 14 after the send timing effect normalizes.
Sub-step 7.1: On the evaluation date, navigate to Campaign Analytics in Instantly and pull the per-variant metrics.
Sub-step 7.2: Confirm both variants have reached minimum sample size. If either variant is below the minimum, do not declare a winner. Extend the test window until minimum sample is reached.
Sub-step 7.3: Record both primary and secondary metrics for both variants in the test log:
Sub-step 7.4: Apply the acceptance threshold. Does the winner exceed the threshold by the required margin? If yes: winner is declared. If no: result is inconclusive — note "insufficient difference" in the test log, archive the test, and plan a new test with a more differentiated Variant B.
Benchmark: A meaningful result meets all three criteria: minimum sample reached, difference exceeds acceptance threshold (5+ pp for open rate, 2+ pp for reply rate), and test ran for full 14-day duration.
Failure mode: Declaring a winner on a 0.5-point open rate difference (32.1% vs 32.6%) as "Variant B is proven." This difference is within normal variance and cannot be reliably reproduced.
Sub-step 8.1: Update the standard template in Instantly with the winning variant's element. If Variant B's subject line won, update the subject line field in your campaign template.
Sub-step 8.2: Archive the losing variant with a note: test date, what was tested, result, why the loser lost (if known). Build a searchable record of what has been tested and rejected, so future practitioners on the team do not re-test the same hypotheses.
Sub-step 8.3: Identify the next test based on priority order (subject line → opening sentence → email length → CTA → personalization depth). If subject line is validated, move to opening sentence. Do not re-test the subject line unless the ICP changes.
Sub-step 8.4: Schedule the next test start date within 5 days of declaring a winner. Momentum in the testing cadence compounds results over time.
Benchmark: After 3 test cycles (one subject line, one opening sentence, one CTA), a campaign has three validated elements. The starting point for campaign 4 is materially stronger than campaign 1. Per Woodpecker's 2025 cold email benchmark study, top-quartile senders who achieve 15–20% reply rates have typically run 4+ test cycles on their primary campaign before reaching that performance level.
Failure mode: Running the same subject line test again rather than moving to the next element in the priority list. The second subject line test produces marginal incremental improvement over the first validated subject line; the first opening sentence test produces much larger improvement because that element has never been optimized.
| Setting | Location in Instantly | Recommended value | Notes |
|---|---|---|---|
| Test split ratio | Sequence editor → Variant settings | 50/50 | Use 80/20 only to protect a strong-performing baseline |
| Min sends per variant (open rate) | Manual tracking | 200 sends | Below this: results are noise |
| Min sends per variant (reply rate) | Manual tracking | 300 sends | Reply tests require larger samples |
| Min sends per variant (CTA test) | Manual tracking | 400 sends | Lowest base rates require most volume |
| Test duration | Calendar reminder | 14 days minimum | Never declare winner before 14 days |
| Winner threshold (open rate) | Manual calculation | 5+ pp absolute difference | A 2pp difference may be variance |
| Winner threshold (reply rate) | Manual calculation | 2+ pp absolute difference | 1pp or less: inconclusive |
| AI auto-optimization | Campaign settings | Off during active test | Auto-winner before min sample invalidates results |
| Variables per test | Campaign editor | 1 (exactly) | Any more: results uninterpretable |
| Daily send volume | Campaign settings | Enough to reach min sample in 10 days | 4-day buffer before 14-day evaluation |
| Test log | External document | One row per test | Record hypothesis, variants, results, winner, next action |
| Inbox assignment | Campaign settings | Identical for both variants | Different inboxes = confounding variable |
After running 8–10 tests, you have a validated component library: 2–3 tested subject line formats with known relative performance, 2 tested opening sentence structures, and 1–2 tested CTA phrasings. This library can be deployed immediately to new campaigns targeting similar ICPs, giving every new campaign a head start over the untested baseline.
Track your test library in a running document with three columns: element type (subject line / opening / CTA), variant copy, test result (win rate, best ICP match). Future campaigns pull from the library instead of starting from scratch.
Most practitioners only test "what might work better." Testing a deliberately simplified or more generic version of your current template against your current version establishes a performance floor: you know how bad performance gets if you remove the elements you have optimized. This is useful for understanding the value of personalization (test: no personalization vs. current personalized template) and the value of specificity (test: generic ICP reference vs. current specific ICP reference).
A winning subject line for VP of Sales in SaaS companies may not transfer to CFOs in manufacturing. Before applying a test result across ICPs, run a transfer test: does Variant A (the validated winner for ICP 1) also win against a representative alternative for ICP 2? If results diverge, segment your templates by ICP rather than using a single universal winner.
Testing subject lines on Campaign A while testing opening sentences on Campaign B (different ICPs) is valid and efficient. The tests do not interfere because they are run on separate audiences. What you cannot do is run two tests within the same campaign and the same sequence step at the same time — that creates confusion about which element is being tested.
Test results are not permanent. A subject line that wins in Q1 (when your ICP is in active planning mode) may underperform in Q3 (when the same ICP is executing projects and has less bandwidth for vendor evaluation). Re-run top-performing variants every 6 months to confirm they still hold up. Flag seasonal patterns in the test log so future test planning accounts for them.
AI-generated personalization is time-consuming and expensive at scale. A direct test of: (A) no personalization vs. (B) role-based personalization vs. (C) AI-generated company-specific personalization — with each variant sent to equal samples of the same ICP — quantifies the exact reply rate lift produced by each additional personalization level. Most teams find that role-based personalization produces 80% of the reply lift of full AI personalization at 20% of the time cost.
After the test runs to completion, navigate to Campaign Analytics and compare the variants:
For subject line tests — primary metric: open rate A difference of 5+ percentage points is meaningful. A 2-point difference (32% vs. 34%) may be within normal variance. A 10-point difference (28% vs. 38%) is a clear signal.
For opening sentence and body copy tests — primary metric: reply rate A difference of 2+ percentage points is meaningful for reply rate (e.g., 7% vs. 9% — a 28% relative improvement). Reply rate differences of less than 1.5 percentage points may not be statistically meaningful at typical cold email sample sizes.
Secondary metric for both: bounce rate If two variants show meaningfully different bounce rates, this suggests the contact distribution between variants was not random. Check for list quality issues before interpreting open or reply rate differences.
Positive reply rate vs. total reply rate Instantly's AI categorization separates replies into: interested, not interested, unsubscribe, out-of-office, and other. For CTA tests, compare positive reply rate (interested only) rather than total reply rate. A CTA that generates more unsubscribe replies will show higher total reply rate but lower positive reply rate — the wrong direction.
Symptom: Variant A and Variant B both produce below-5% reply rates. Testing feels pointless because neither is performing.
Cause: This is an ICP or list quality problem, not a testing problem. The audience is wrong, the list is poor quality, or the core value proposition does not land for this segment. Tweaking subject lines and opening sentences cannot fix an ICP mismatch.
Fix: Pause both variants. Review the audience definition. Confirm that the contacts match the ICP you designed the campaign for. Check bounce rates — if above 5%, the list quality is the issue; replace contacts with pre-verified data from Quarvio. Do not run another A/B test on this audience until reply rate for a baseline single-variant campaign exceeds 5%.
Symptom: Campaign Analytics shows aggregate metrics but no per-variant breakdown.
Cause: The A/B test was not configured at the sequence step level, or variants were not saved correctly before launching. The analytics view only shows per-variant data if the test was properly set up in the sequence editor.
Fix: Navigate to the campaign sequence editor and confirm that Variant B exists as a separate variant on the specific step being tested (not as a separate campaign). If the test was not properly configured, pause the campaign, reconfigure the split in the sequence editor, and resume.
Symptom: After 10 days, Variant A has 280 sends and Variant B has 120 sends, despite a 50/50 split setting.
Cause: Instantly's distribution can front-load contacts to one variant if daily send volume is low and the first batch of contacts was assigned to Variant A by chance. At low send volumes, early randomization can look uneven.
Fix: Check the daily send totals over the test period. If daily distribution is approximately equal (Variant A: 28/day, Variant B: 22/day), the cumulative imbalance is within normal randomization variance and will narrow over time. If daily sends are consistently biased to one variant, check that the split setting saved correctly — navigate back to the sequence editor and confirm the 50/50 setting is active.
Symptom: After 7 days, Variant A leads 36% to 29% open rate. After 14 days, Variant B leads 33% to 31%.
Cause: This is normal and exactly why tests must run for a full 14 days. Week 1 opens cluster heavily on whatever day the campaign launched — if it launched Monday, week 1 has more Monday opens (higher than average for many ICPs) and fewer Friday opens (lower than average). Week 2 normalizes across the full week distribution.
Fix: Trust the 14-day result, not the 7-day result. Document both data points in the test log. If you see consistent reversal patterns across multiple tests, your ICP may have strong day-of-week effects that require segment-level analysis.
Symptom: After 14 days, Variant A has 140 sends and Variant B has 140 sends — both below the 200-per-variant minimum.
Cause: The available contact pool was too small for the test volume required. You either overestimated the available contacts or underestimated the required sample size.
Fix: Do not declare a winner. The test is inconclusive. Top up the contact list with additional verified contacts from Quarvio that match the same ICP definition. Do not expand the ICP definition just to increase contact count — adding different audiences to reach minimum sample invalidates the test. Resume the campaign with the additional contacts and extend the test window.
Symptom: After reviewing Variant A and Variant B, you realize the subject line and the opening sentence both differ between variants.
Cause: The variants were edited independently and a secondary change was made accidentally, or the test was not cross-checked before launch.
Fix: The test is invalid and cannot be attributed to either element. Archive the test with note "invalid — multiple variables changed." Do not use the result to update any campaign element. Run a clean test with one variable changed.
Symptom: Variant B shows 12% total reply rate vs. Variant A's 9%. But when you look at reply categories, Variant B has 4% positive (interested) and 8% negative/unsubscribes. Variant A has 7% positive and 2% negative/unsubscribes.
Cause: The CTA or message tone in Variant B is generating strong reaction — but mainly negative. It is getting people to respond to say they are not interested rather than to engage positively.
Fix: Use Instantly's AI categorization to compare positive reply rate (interested only) rather than total reply rate for CTA and message body tests. For this example, Variant A wins on positive reply rate (7% vs. 4%) despite losing on total reply rate. Update the template with Variant A.
Symptom: Instantly pauses Variant B after 7 days with 80 sends each because Variant A is leading. You wanted to run the test to completion.
Cause: Instantly's AI optimization feature auto-declares winners to maximize campaign performance. This is useful for production campaigns but interferes with controlled A/B tests where you want to run the full sample before evaluating.
Fix: Disable AI auto-optimization in the campaign settings before running controlled A/B tests. The setting is typically found in Campaign Settings → Optimization. For production campaigns where you want Instantly to manage performance automatically, leave it enabled. For explicit testing, disable it.
Instantly reviews on G2 consistently identify A/B testing as a feature that practitioners wish they had started using earlier, with multiple reviewers noting that the first validated subject line test alone produced a 5–10 percentage point open rate improvement.
Woodpecker's 2025 cold email benchmark study establishes that the 8.5% average reply rate across all senders is an average across tested and untested campaigns; practitioners who use systematic testing consistently land in the 12–20% range for their highest-performing campaigns.
"I ran four subject line tests over two months. Each test eliminated a weaker approach and validated a stronger one. My open rate went from 24% to 41% over those eight weeks. Not from guessing — from testing. The Instantly A/B feature took maybe 10 minutes to configure. The discipline of running it correctly is the actual skill."
— Verified G2 reviewer, outbound marketing lead, B2B SaaS, Instantly reviews on G2
"The biggest mistake I see is people testing two slightly different phrasings of the same idea and calling the winner 'validated.' A real test pits two meaningfully different approaches against each other. If both variants are variations on 'quick question', the test teaches you nothing. Make the variants genuinely different."
— Verified G2 reviewer, cold email consultant, Instantly reviews on G2
| Need | Tool | Notes |
|---|---|---|
| Verified B2B contacts | Quarvio | One-time purchase, no subscription |
| Email inboxes | Inframail | Microsoft 365 inboxes, auto DNS |
| Cold email sending | Instantly | Sequences, warm-up, A/B testing, reply tracking |
| LinkedIn outreach | Aimfox | Connection campaigns, Unibox |
How do I run an A/B test in Instantly?
In Instantly, create a new campaign and in the sequence editor, add a second variant to the step you want to test. Change only the one element you are testing in Variant B (subject line, opening sentence, CTA, or email length). Set the split percentage (50/50 recommended), confirm only one element differs between Variant A and Variant B, and set a minimum test duration of 14 days. Navigate to Campaign Analytics to compare variant performance once the test completes and both variants have reached minimum sample size.
How do I set up a split test in Instantly step by step?
Navigate to Campaigns → New Campaign → sequence editor. Add your base email as Variant A. Click the variant or A/B test option on the sequence step you are testing and create Variant B. Change only the one test element in Variant B. Set the split ratio (50/50 for most tests). Configure campaign sending settings. Launch the campaign. Monitor bounce rate in the first 24 hours; do not check open or reply rate results until minimum sample is reached at the 14-day mark. Declare a winner based on the threshold defined in your hypothesis.
What sample size do I need for a cold email A/B test in Instantly?
A minimum of 200 sends per variant for open rate tests (subject line); 300+ per variant for reply rate tests (opening sentence, email length, CTA); 400+ per variant for CTA phrasing tests. Below these thresholds, the variance from random factors (day of week, prospect timing, chance) is large enough to produce false positives. If your contact pool is smaller than the required minimum, run a single-variant campaign instead of splitting.
How long should I run an A/B test in cold email?
A minimum of 14 days. Cold email performance is affected by day-of-week sends, which only normalize after two complete weeks. A variant that leads after 7 days may trail after 14 days once the weekday effect averages out. Do not declare a winner before the 14-day mark, and do not declare a winner before both variants have reached their minimum sample size, even if that takes longer than 14 days.
What should I A/B test first in cold email?
Subject line. Open rate is the gating metric for everything else in cold email — if the email is not opened, nothing else matters. A subject line test directly answers "what gets this specific audience to open?" and the result applies immediately to all subsequent campaigns targeting that audience. Run a question-format vs. statement-format subject line test as your first test, then move to opening sentence once subject line is validated.
How do I know if my cold email A/B test result is statistically significant?
For open rate tests: a 5+ percentage point absolute difference at 200+ sends per variant and after 14 days is a meaningful result. For reply rate tests: a 2+ percentage point absolute difference at 300+ sends per variant and after 14 days. Below these thresholds, the difference may be random variance rather than a real signal. Cold email cannot easily reach formal 95% statistical confidence (which would require 1,000+ sends per variant for typical reply rate improvements) — but the practical thresholds above are sufficient for confident template updating.
Can I run multiple A/B tests at the same time in Instantly?
You can run tests across different campaigns simultaneously — for example, a subject line test on Campaign A (targeting VP of Sales in SaaS) and an opening sentence test on Campaign B (targeting HR Directors in manufacturing). These tests do not interfere because they run on different audiences. What you cannot do is run two tests within the same campaign and the same sequence step simultaneously. One test per campaign element at a time.
Why is my Instantly A/B test not showing variant breakdown in analytics?
The per-variant breakdown only appears if the A/B test was correctly configured at the sequence step level in the sequence editor before the campaign was launched. If you see only aggregate metrics, the test was likely not saved correctly. Navigate to the campaign sequence editor, confirm that Variant B exists as a distinct variant on the specific step being tested, and check that the split ratio is set. If the campaign has already sent, you will need to reconfigure on a new or reset campaign.
How do I read A/B test results in Instantly analytics?
Navigate to Campaign Analytics and select the per-variant view. For each variant, record: open rate, reply rate (total), reply rate by category (positive/interested, unsubscribe, out-of-office), and bounce rate. For subject line tests, compare open rates. For body copy and CTA tests, compare positive reply rates. A variant wins when it exceeds the acceptance threshold you set in your hypothesis (5+ pp open rate difference or 2+ pp positive reply rate difference).
What is the difference between Variant A and Variant B in Instantly?
Variant A is the control — the original or current version of the email element you are testing. Variant B is the test — the changed version. In a properly structured A/B test, Variant A and Variant B differ in exactly one element (subject line, opening sentence, email length, or CTA). Everything else should be identical. Instantly distributes your contact list between the two variants at the split ratio you configure and tracks metrics separately for each.
Should I measure open rate or reply rate for a cold email A/B test?
It depends on what you are testing. Subject line tests — measure open rate, because the subject line only affects whether the email gets opened. Opening sentence, email length, and CTA tests — measure reply rate (specifically positive reply rate after AI categorization), because these elements only affect what happens after the email is opened. Never use reply rate as the primary metric for a subject line test; a subject line can increase opens while a worse body copy reduces replies, making the subject line look like it underperformed.
How do I apply A/B test findings to future campaigns in Instantly?
After declaring a winner, update the element in your campaign template with the winning variant's copy. Archive the losing variant in your test log with the result and a brief note on why it underperformed. Move to the next element in the testing priority order (subject line → opening sentence → email length → CTA → personalization). After 3–4 test cycles, you have a campaign template where every major element has been validated against alternatives. Every new campaign built from this template starts from a validated baseline.
A/B testing works best with large enough samples
Meaningful test results require 200+ contacts per variant — and those contacts need to be verified deliverable, or high bounce rates will contaminate your open and reply rate data. Quarvio delivers verified B2B contact lists by job title, industry, and company size so you have enough qualified contacts to run valid tests without exhausting your audience. One-time purchase, credits valid for 12 months, no subscription.
