Issue

When reading data from PostgreSQL tables in a Semarchy xDI mapping, the execution may consume excessive memory and eventually fail with an error similar to:

java.lang.OutOfMemoryError: Java heap space

This issue is typically observed on large source tables and may significantly increase Runtime memory consumption.

Cause

The PostgreSQL JDBC driver may retrieve large result sets into memory depending on the JDBC configuration and execution mode. When the driver does not stream data efficiently, the Runtime may need to allocate large amounts of memory to process the result set.

Contributing factors include:

  • Large source datasets.
  • SQL Fetch Size not configured appropriately.
  • Non-transactional execution mode.
  • Insufficient Runtime heap allocation.

Solution

  1. Review the mapping and identify the PostgreSQL source datastore involved.
  2. Configure an appropriate SQL Fetch Size value in the mapping template parameters.
  3. When supported by the template, enable Transactional Mode so that the PostgreSQL JDBC driver can stream records using the configured fetch size.
  4. For large datasets, avoid unnecessarily large fetch sizes that could increase memory consumption.
  5. If required, increase the Runtime JVM memory allocation using the Runtime startup options:
XDI_RUNTIME_OPTS="-Xms2g -Xmx8g"

or

XDI_COMMAND_OPTS="-Xms2g -Xmx8g"
  1. Restart the Runtime after applying the configuration changes.

Verification

  • Monitor Runtime memory consumption during execution.
  • Validate that the process completes without heap space errors.
  • Review execution logs for improved fetch behavior and reduced memory usage.

Tip: Start by tuning the SQL Fetch Size before increasing JVM memory. Excessive memory consumption is often caused by inefficient result-set retrieval rather than insufficient heap allocation.