Building an A/B Testing & Feature Flag Platform from Scratch

Most teams reach for LaunchDarkly or Statsig and call it a day. I decided to build my own — not because off-the-shelf tools are bad, but because understanding what's happening under the hood is the difference between a developer and an engineer.

This is a breakdown of how I architected and built a full-stack A/B testing and feature flag platform, from the evaluation algorithm to the SDK, caching layer, and event pipeline.

Why Build Your Own?

Before touching code, the "why" matters. Commercial platforms are:

Expensive at scale — pricing is often per-seat or per-evaluation
Black boxes — you don't control the evaluation logic
Generic — can't customize the data model to your domain

Building your own forces you to confront the real complexity: how do you ensure consistent variant assignment? How do you evaluate flags without a database round-trip on every request? How do you handle millions of events without dropping data?

System Architecture Overview

The platform has three major pieces:

NestJS Backend — flag management, evaluation API, experiment tracking
Node.js SDK — client-side evaluation with caching and event buffering
Next.js Dashboard — visual flag management with real-time stats

code

┌─────────────────────┐     Bearer <env-api-key>     ┌──────────────────────┐
│    Your App / SDK   │ ────────────────────────────▶ │   NestJS REST API    │
└─────────────────────┘                               │                      │
                                                      │  ┌────────────────┐  │
                                                      │  │  SdkService    │  │
                                                      │  │  (Evaluator)   │  │
                                                      │  └───────┬────────┘  │
                                                      │          │            │
                                                      │  ┌───────▼────────┐  │
                                                      │  │  Redis Cache   │  │
                                                      │  └───────┬────────┘  │
                                                      │          │            │
                                                      │  ┌───────▼────────┐  │
                                                      │  │   PostgreSQL   │  │
                                                      │  └────────────────┘  │
                                                      └──────────────────────┘

The Data Model

The entity hierarchy was the first real design challenge. Everything flows from Organization → Project → Environment.

code

Organization
  └── Project (has a short projectId like "my-app-A3F2B")
        ├── Environment (development | staging | production)
        ├── ApiKey      (scoped per environment)
        └── FeatureFlag
              ├── Variant[]  (name, weight: 0–100, value)
              └── FeatureFlagEnvironment  (per-env config: enabled, rollout%, targeting rules)

The key insight: flags and their environments are separated. A flag can be enabled: true globally but enabled: false in production, and rolled out to only 20% of staging. This separation keeps the data model clean and eliminates conditional spaghetti.

Feature Key Generation

Every flag gets a unique, human-readable key like dark-mod-A1B2C3D4. The algorithm:

typescript

function generateFeatureKey(name: string): string {
  const base = name
    .toLowerCase()
    .replace(/[^a-z0-9]+/g, '-')
    .slice(0, 8);                          // first 8 chars of slug
  const suffix = randomBytes(4)
    .toString('hex')
    .toUpperCase();                        // e.g. "A1B2C3D4"
  return `${base}-${suffix}`;
}

If there's a collision (rare, but possible), the service loops and regenerates. This is the key your SDK uses — not a UUID, so it's readable in code.

The Core Algorithm: Deterministic Hashing

This is the most important part. Every user must always get the same variant for a given flag, across any server, any language, any time. If a user sees "Variant B" and refreshes, they must see "Variant B" again. This is called sticky bucketing.

The solution: deterministic hashing.

typescript

private getBucket(userId: string, flagKey: string, salt?: string): number {
  const input = `${userId}${flagKey}${salt ?? ''}`;
  const hex = createHash('sha256').update(input).digest('hex').slice(0, 8);
  return parseInt(hex, 16) % 100; // deterministic 0–99 bucket
}

SHA-256 on userId + flagKey + salt always produces the same output. Taking % 100 gives a bucket from 0–99. The hashSalt per flag means the same user gets a different bucket for different flags — eliminating correlation between experiments.

Variant Assignment via Cumulative Weights

Once we have a bucket (0–99), we walk the sorted variants and find where the bucket falls:

typescript

private assignVariant(variants: Variant[], bucket: number): Variant {
  const sorted = [...variants].sort((a, b) => a.name.localeCompare(b.name));

  let cumulative = 0;
  for (const variant of sorted) {
    cumulative += Number(variant.weight);
    if (bucket < cumulative) return variant;
  }

  return sorted[0]; // fallback
}

For a 33/33/34 split across three variants:

Bucket 0–32 → Variant A
Bucket 33–65 → Variant B
Bucket 66–99 → Variant C

The sort-by-name before cumulative summation ensures a stable ordering regardless of insertion order in the database.

The 8-Step Evaluation Pipeline

Every flag evaluation on the server goes through a strict 8-step waterfall:

code

1. Cache Aside  →  Redis hit? Return immediately.
2. Kill Switch  →  flag.enabled === false? Return GLOBALLY_DISABLED.
3. Env Override →  ffEnv.enabled === false? Return ENV_DISABLED.
4. Targeting    →  targetingRules don't match? Return TARGETING_MISS.
5. Rollout      →  bucket >= rolloutPercentage? Return OUT_OF_ROLLOUT.
6. Variant      →  assignVariant() → pick winner.
7. Log          →  createAssignmentLog() for experiments.
8. Exposure     →  trackEvent('$exposure') for analytics.

Each step returns early with a typed reason string (GLOBALLY_DISABLED, ENV_DISABLED, TARGETING_MISS, etc.), which is surfaced back to the SDK. This makes debugging trivial — you always know why a user got a certain result.

The Targeting Engine

Targeting rules let you gate flags to specific audiences: "users in the US with a cart value > $50 and a Pro plan". I modeled this as TargetingRules with an OR-of-ANDs structure:

json

{
  "groups": [
    {
      "conditions": [
        { "attribute": "country", "operator": "in", "values": ["US", "CA"] },
        { "attribute": "plan",    "operator": "equals", "values": ["pro"] }
      ]
    }
  ]
}

Each group is AND'd internally, and groups are OR'd together. The engine:

typescript

private evaluateTargetingRules(rules, attributes): boolean {
  // OR across groups — any matching group = targeted
  return rules.groups.some(group =>
    // AND across conditions — all must pass within a group
    group.conditions.every(c => this.evaluateCondition(c, attributes))
  );
}

Supported operators: in, not_in, equals, not_equals, contains, starts_with, ends_with, gt, gte, lt, lte, exists, not_exists.

This covers virtually every audience segmentation need without requiring a query language parser.

Caching with Redis (Cache-Aside Pattern)

A database round-trip per flag evaluation is a non-starter at scale. The solution is Redis with the cache-aside pattern:

typescript

const config = await this.redisService.cacheAside(
  `flag:${flagKey}:env:${envId}`,
  async () => {
    // Fetch flag + experiment + ffEnv in parallel
    const [experiment, ffEnv] = await Promise.all([...]);
    return { flag, experiment, ffEnv };
  },
  30, // TTL: 30 seconds
);

The cache key is flag:<flagKey>:env:<envId>. On a cache hit, zero DB queries run. On a miss, we fetch and write back with a 30-second TTL. Flag changes propagate within 30 seconds — acceptable for most feature flag use cases.

For immediate propagation (e.g., emergency rollback), the cache key can be explicitly invalidated via redisService.del(cacheKey) from the write path.

The SDK: Client-Side Architecture

The Node.js SDK wraps evaluation and event tracking in a clean, non-throwing API:

typescript

const sdk = new ABTestingSDK({ apiKey: process.env.AB_KEY });

// Never throws — returns disabled result on error
const { enabled, variant } = await sdk.evaluate('dark-mod-A1B2C3D4', {
  userId: 'usr_123',
  userAttributes: { country: 'US', plan: 'pro' }
});

Three layers work together:

1. LocalCache — In-Memory TTL Cache

typescript

get(key: string): T | null {
  const entry = this.store.get(key);
  if (!entry) return null;
  if (Date.now() > entry.expiresAt) {
    this.store.delete(key); // lazy eviction
    return null;
  }
  return entry.value;
}

A simple Map<string, { value, expiresAt }> with lazy eviction. The cache key is flagKey:userId, so different users always get their own cached result.

2. FlagEvaluator — Network Layer with Caching

typescript

async evaluate(flagKey: string, options: EvaluateOptions): Promise<FlagResult> {
  const cacheKey = `${flagKey}:${options.userId}`;
  const cached = this.cache.get(cacheKey);
  if (cached) return cached; // 0ms

  const result = await this.http.post('/sdk/evaluate', { ... });
  this.cache.set(cacheKey, result);
  return result;
}

3. EventQueue — Batched, Fault-Tolerant Tracking

This was the most interesting piece to design. Naive event tracking fires one HTTP request per event — terrible for high-throughput apps.

typescript

push(event: TrackOptions): void {
  this.queue.push(event);
  if (this.queue.length >= this.batchSize) {
    this.flush(); // flush early if batch is full
  }
}

async flush(): Promise<void> {
  if (this.queue.length === 0) return;
  // Atomically drain before the async call — prevents double-send
  const batch = this.queue.splice(0, this.queue.length);
  try {
    await this.http.post('/sdk/track/batch', { events: batch });
  } catch (err) {
    // Put events back at the front for retry on next flush
    this.queue.unshift(...batch);
  }
}

The key design choice: drain atomically before the async call. If you drained after, a concurrent push() during the in-flight request could lose events. By splicing first, concurrent pushes go into the array after position 0, and if the flush fails, we unshift the failed batch back to the front — preserving order.

Events are flushed on a timer (flushIntervalMs, default 10s) or when batchSize (default 100) is reached. destroy() triggers a best-effort final flush before process exit.

What I Learned

1. Deterministic hashing is foundational. Get this wrong and your experiments are worthless — users see different variants on every request and you can't draw any conclusions.

2. The cache-aside pattern at the right layer. Caching at the server level (Redis) handles scale. Caching at the SDK level (LocalCache) handles latency. Both are necessary.

3. Design for failure, not the happy path. The SDK never throws. The event queue retries on failure. The evaluation pipeline returns a reason on every exit path. Resilience is a first-class concern.

4. Data model correctness unlocks everything. Separating FeatureFlag from FeatureFlagEnvironment made per-env toggles, rollout percentages, and targeting rules trivial to implement. A flat schema would have made all of this a mess.

What's Next

Statistical significance engine — Bayesian and frequentist analysis on experiment results
Real-time streaming — Server-Sent Events for instant flag updates without SDK polling
Guardrail metrics — Auto-rollback experiments when error rates or latencies breach thresholds
Multi-language SDKs — Python, Go, and a React hooks wrapper

The full source is available on my GitHub. If you're building something similar, start with the hashing algorithm — get that right and everything else falls into place.

Happy experimenting. 🧪