Data flow

A SQIP data flow begins when a producer or source connector writes records to a topic. One or more consumers or sink connectors can then process those records independently.

Basic flow

Producer or source connector
          ↓
        Topic
          ↓
Consumer or sink connector

Producers

A producer sends records to a topic. It can be:

Your application using a namespace-scoped API key.
An HTTP REST, FTPS, or PostgreSQL source connector.

Consumers

A consumer reads topic records at its own pace. It can be:

Your application using a namespace-scoped API key.
An HTTP REST or PostgreSQL sink connector.

Multiple consumers can read the same topic for different purposes without changing the original stream.

Records and keys

A record contains the event value and may include a key. Stable keys help keep related records together and make retries safer for destinations that update existing rows.

Examples of useful record keys include:

order_id
customer_id
device_id

SQIP connectors prioritize reliable delivery. A record may be delivered again when a connector retries after a timeout or interruption. Design destinations to tolerate duplicates, especially for HTTP calls and database writes.

Tip

Prefer stable record keys and upsert behavior when the destination supports them.

Schemas as data contracts

A schema helps producers and consumers agree on record structure. Compatibility rules determine whether a new schema version can safely coexist with older applications.

Use a schema when:

Multiple teams share a topic.
Records change over time.
Consumers require predictable fields and types.
You want incompatible changes rejected before they reach production.

Choosing between API keys and connectors