Prompt Management

Flow's prompt management lets you version, test, and roll out system prompts centrally — without redeploying your application. You can A/B test prompt variants, do canary rollouts, and use template variables to personalize prompts at runtime.

Typical Workflow

Most teams follow this progression:

1. Connect to the Gateway

Point your application at Flow by changing the base_url. Your app works identically — all messages including your system prompt pass through untouched.

from openai import OpenAI

client = OpenAI(
    api_key="dh_your_key",
    base_url="https://gateway.datahippo.io/v1"
)

At this point, prompt management is not active. Your project defaults to explicit mode.

2. Build Your Prompts

Create prompt configs and versions in the DataHippo UI. Each version can include:

• System prompt (with optional template variables)
• Model override (e.g., pin a prompt to gpt-4o)
• Temperature and max_tokens
• Tools / function definitions
• Response format (for structured output / JSON mode)

You can create as many versions as you want. They are stored and versioned but have no effect on production traffic while prompt mode is explicit and you are not sending a prompt_config in your requests.

3. Test with Explicit Opt-In

While still in explicit mode, you can test a specific prompt config on individual requests without affecting the rest of your traffic:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"prompt_config": "my-prompt-config"}
)

Or via header:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={"x-datahippo-prompt-config": "my-prompt-config"}
)

This is useful for canary testing — you can route a subset of your traffic through a managed prompt while the rest continues to use your application's built-in prompt.

4. Switch to Auto Mode

Once you're confident in your managed prompts, switch the project's prompt mode to auto. Now, if the project has an active rollout, the managed prompt applies to all requests automatically — no code changes needed.

Prompt Modes

Mode	Behavior
explicit	Managed prompts apply only when the request includes a prompt_config (header or body). Active rollouts are ignored for requests without it. This is the default.
auto	Active rollouts apply to all requests automatically. Individual requests can still specify a prompt_config to override the rollout.

How Managed Prompts Are Applied

When a managed prompt is resolved (either through explicit opt-in or auto mode), it is prepended to your existing system prompt — not replaced.

• If your request has a system message, the managed prompt is inserted before it: "{managed}\n\n{yours}".
• If your request has no system message, a new one is created with the managed prompt.
• All other messages (user, assistant, tool) are never modified.

Rollouts

A rollout lets you gradually shift traffic from a baseline prompt version to a target version. Rollouts support:

• Staged rollout: Define stages with increasing target weight (e.g., 10% → 50% → 100%).
• A/B testing: Split traffic between baseline and target, then compare metrics.

Allocation Strategies

Strategy	Description
random	Each request is randomly assigned to baseline or target based on weight.
user_sticky	The same user always gets the same variant. Requires x-datahippo-user-id header.
session_sticky	The same session always gets the same variant. Requires x-datahippo-session-id header.

Rollout Statuses

Status	Description
pending	Created but not yet active.
running	Actively routing traffic.
paused	Temporarily stopped; requests fall through without a managed prompt.
completed	Finished; the target version is now the default.
rolled_back	Reverted to baseline.

Force a Variant (Debugging)

During a rollout, you can force a specific variant for testing:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={"x-datahippo-force-variant": "target"}
)

Values: "target" or "baseline".

Template Variables

Managed prompts support Handlebars-style template variables. Define variables in the prompt version, then pass values at runtime:

Prompt template:

You are a helpful assistant for {{company_name}}.
The user's name is {{user_name}}.

Pass variables via headers:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={
        "x-datahippo-var-company-name": "Acme Corp",
        "x-datahippo-var-user-name": "Alice"
    }
)

Or via request body (no 255-character limit):

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "prompt_variables": {
            "company_name": "Acme Corp",
            "user_name": "Alice"
        }
    }
)

Header names are normalized: x-datahippo-var-company-name becomes the variable company_name. Header variables take precedence over body variables when the same key appears in both.

Variable Validation

Each variable can define a schema with constraints:

• required — must be provided (or have a default).
• default — value used when the variable is not supplied.
• var_type — string, number, boolean, or enum.
• values — allowed values for enum-typed variables.
• max_chars — maximum character length.
• min / max — numeric bounds.

If a variable fails validation, the request returns a 400 error with details about which variable and what constraint failed.

Summary Table

Scenario	Managed prompt applied?	Your system prompt
explicit mode, no prompt_config	No	Preserved exactly
explicit mode, with prompt_config	Yes	Prepended by managed prompt
auto mode, no active rollout	No	Preserved exactly
auto mode, active rollout	Yes	Prepended by managed prompt