Skip to main content
Last updated on

PostgreSQL Schema Change Sync

PostgreSQL Schema Change Sync automatically applies upstream PostgreSQL column changes to Doris during continuous load. This capability applies only to PostgreSQL CDC with Auto Table Creation. It is not supported by PostgreSQL CDC with SQL Mapping.

tip

PostgreSQL Schema Change Sync is supported from Doris 4.1.0.

Supported Schema Changes

PostgreSQL operationDoris behavior
ADD COLUMNAdds a column with the same name and a type defined by the PostgreSQL Data Type Mapping. DEFAULT and NOT NULL constraints are not copied, and historical rows are not backfilled. Subsequent rows use the actual values written by PostgreSQL to WAL.
DROP COLUMNDrops the column with the same name.

Considerations

  • A PostgreSQL schema change does not propagate immediately when only the DDL is executed. Doris detects and applies the new schema after the changed table receives a subsequent INSERT, UPDATE, or DELETE.
  • Automatic schema change synchronization is not supported during the full snapshot phase. Perform upstream schema changes after the full snapshot completes and the job enters the incremental phase.
  • If an added column already exists or a dropped column does not exist, Doris skips the operation so that a retry does not fail the job.
  • If one schema change adds and drops columns at the same time, Doris treats it as a possible column rename and does not automatically change the target table. This prevents accidental data loss.
  • Column renames, column type changes, DEFAULT changes, and NULL / NOT NULL constraint changes are not synchronized automatically. Pause the continuous load job, change the Doris target table manually, verify that both schemas are compatible, and then resume the job.
  • You can disable automatic schema change synchronization with the Job property schema_change_enabled (default true, supported since version 4.1.4). The cdc_stream() table function (SQL Mapping Sync) always forces this property to false.
Behavior change (4.1.4)

Since version 4.1.4, PostgreSQL schema change detection is driven by Relation events:

  • Only ADD COLUMN and DROP COLUMN are recognized.
  • A change that adds and drops columns at the same time (possibly a RENAME) is skipped, and the target table is left unchanged.
  • Column type changes are skipped.
  • An added column no longer carries the DEFAULT value of the upstream column.
  • This capability applies only to the Auto Table Creation Sync (at-least-once) path. The SQL Mapping (TVF / exactly-once) path does not support it.