Message Archiving and Tier 2 Storage
Message Archiving and Tier 2 Storage
Connexion v26 can move older message data out of the hot queue database and into lower-cost Tier 2 storage while keeping that data queryable from Connexion.
Internally this capability is also called hot storage (the repository database) and warm or cold storage. The two names refer to the same feature.
How it works
Connexion queue repositories use two storage tiers:
Tier | Purpose |
|---|---|
Tier 1 (hot storage) | The active SQL Server or PostgreSQL queue database containing current and recent message partitions used for operational queue work |
Tier 2 (warm storage) | Historical message partitions stored as Parquet files on a filesystem or cloud object store |
Once a partition ages beyond the configured Tier 1 retention window and contains no queued or delayed work that would block archiving, Connexion can:
Mark the partition ready for archiving.
Copy its messages and related data into Parquet files.
Verify the archived data.
Remove the verified partition from the hot database.
Continue routing message queries across both storage tiers.
Messages remain continuously accessible during archiving. Connexion keeps the source partition in Tier 1 until its Parquet copy has been written to Tier 2 and verified. Only then is the partition dropped from the hot database.
Operational impact
Pruning drops from hours to minutes. On large systems, the pruning stage of database maintenance could previously spend many hours deleting historical rows. With partition-based Tier 2 archiving, maintenance can drop a verified hot partition instead, reducing that stage to minutes.
The hot database stays smaller and less expensive. Tier 1 carries only the operational window, reducing storage, I/O, backup, and maintenance overhead. This can materially reduce database cost when using cloud or managed database providers.
The hot database therefore stays focused on current operational work instead of carrying the entire historical retention period.
Historical messages remain queryable
Archived data is not an offline backup that must be restored before use.
Connexion uses Parquet files and transparently combines results from Tier 1 and Tier 2. Normal UI and MCP message queries can cross the hot/cold boundary, including paging through results from both tiers.
Messages returned from Tier 2 are marked with a blue storage indicator in the leftmost column. Hover over the indicator to display the path of the Parquet file containing that message.
Apart from this source indicator, the normal queue query and inspection workflow does not change. Connexion routes each request to the appropriate tier and aggregates messages across both storage tiers.
Queries against object storage may have different latency from queries against the hot database, but users do not need to manually locate and restore an archive file before inspecting a historical message.
Storage targets
v26 supports the following tier-2 targets:
Local or shared filesystem storage.
Amazon S3.
Azure Blob Storage.
Google Cloud Storage.
Tier 2 configuration hierarchy
Tier 2 behavior flows from the repository to the group and finally to the individual queue. Each level inherits its parent by default, while more specific levels can override the inherited behavior.
Level | Purpose |
|---|---|
Repository | Defines whether Tier 2 storage is enabled and supplies the default inclusion behavior, archived-data retention, and storage target for groups and queues using that repository. |
Group | Inherits the repository configuration by default. A group can override the default inclusion behavior, archived-data retention, or storage target. A group-level force-disable setting excludes every queue in that group. |
Queue | Inherits the effective repository and group behavior by default. An individual queue can explicitly opt in to or out of Tier 2 storage unless Tier 2 has been disabled at the repository or group level. |
When Tier 2 is enabled, Connexion evaluates the most specific applicable setting: an explicit queue choice takes precedence over the group default, and the group default takes precedence over the repository default. Repository-level disablement and group-level force-disablement are hard exclusions.
Scope and retention
In v26, message retention is configured per group, not per queue as it was in v16. Every queue in a group follows the group retention policy.
PostgreSQL repositories can assign different Tier 1 retention values to different groups. SQL Server repositories use one Tier 1 retention value for all groups sharing that repository. Per-queue controls determine Tier 2 inclusion or exclusion; they do not establish a separate message-retention duration.
Only partitions outside the Tier 1 retention window and with no queued or delayed work that would block archiving are eligible. Current operational work remains in Tier 1.
Security and PHI
Tier-2 files may contain PHI and must be governed like the live queue database.
Restrict filesystem, bucket, container, and cloud-identity access.
Use the cloud provider's encryption-at-rest controls where available.
Connexion can additionally encrypt Parquet message/data files using its configured archive encryption option.
Align archive retention and deletion with organizational and regulatory policy.
Verify that backup, replication, lifecycle, and support-access policies also cover the tier-2 target.
SQL Server and PostgreSQL
Message archiving is available for the partitioned SQL Server and PostgreSQL queue implementations. Both route eligible historical partitions through the same Parquet-based storage model and expose archived results through the normal query experience.
What this enables
Longer message retention without allowing the operational database to grow without bound.
Lower-cost storage for older messages.
Cloud-native retention using existing object-storage platforms.
Continued investigation and audit access to historical messages.
Cleaner separation between current queue processing and historical data.