Batch deleted after inserting records from Aperture to Databricks

Uma
Uma Member
in General

Hi Team,

I am trying to insert the records from Aperture to Datbricks through JDBC driver and as per the log the records has been inserted. But the batch has been deleted .due to that the output table has no records.

2026-02-11 13:29:31,514 INFO c.e.d.w.e.o.i.JDBCExportIOIterator [workpool-server-lowmem-executor-closer-382] EXPORT COMPLETED IN 1463246ms

2026-02-11 13:29:31,515 INFO c.e.d.w.e.o.i.JDBCExportIOIterator [workpool-server-lowmem-executor-closer-382] BATCH SIZE: 1

2026-02-11 13:29:31,515 INFO c.e.d.w.e.o.i.JDBCExportIOIterator [workpool-server-lowmem-executor-closer-382] INSERT: 500

2026-02-11 13:29:31,569 INFO c.e.d.r.d.DatasetStore [post-executor-779] Deleted batch with version number '1' from dataset with id '5e1aae44-92ee-4654-8582-9d5c6a3280b7'.

Could you please let me know why the batch is getting deleted after inserting the records from aperture to Databricks.

Regards

Uma

Comments

  • Henry Simms
    Henry Simms Administrator
    edited February 12

    To confirm, is the batch of data deleted from Aperture's Dataset?
    If so, do you have automatic batch deletion on execution configured - this would be set in the dataset details:

    image.png

    and in the Source step in the workflow:

    image.png

    It will have the effect of deleting batches that have been processed via a workflow execution.

    However, this setting would not apply to data exported to and External System like Databricks. If data is successfully exported (it looks like it is being) but the target table in Databricks is blank, that sounds like something that may be happening Databricks side.

    Alternatively, you may have some other form of automatic batch deletion configured, eg a maximum batches to retain setting (Dataset settings → Automatically delete older batches. Keep only the latest x)

    The only other thing to check would be the Post SQL in the Export step, to make sure it's not calling some follow-on SQL to cause the table data to be truncated.

Categories