We need to load large files (approx. 10,000 lines) through the xDM user interface using an import action.
For traceability, we would like to store the source file's line number in a dedicated column (e.g., "line_number") within the golden table. Is this possible to achieve natively, or does it require a custom workaround?
Thanks for your feedback,
Jean-Christophe
Best Answer
A
Alexia SIMOND
said
6 months ago
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
If you store this value in the golden table, it becomes business data from xDM’s point of view, so you should define clearly which line number should survive when multiple source records consolidate into one golden record.
For fuzzy-matched entities in particular, several source/master records can contribute to the same golden record, so a single line_number in GD may become ambiguous unless you define a survivorship rule or a dedicated audit design.
1 Comment
A
Alexia SIMOND
said
6 months ago
Answer
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
If you store this value in the golden table, it becomes business data from xDM’s point of view, so you should define clearly which line number should survive when multiple source records consolidate into one golden record.
For fuzzy-matched entities in particular, several source/master records can contribute to the same golden record, so a single line_number in GD may become ambiguous unless you define a survivorship rule or a dedicated audit design.
Jean-Christophe Blanchard
Hello,
We need to load large files (approx. 10,000 lines) through the xDM user interface using an import action.
For traceability, we would like to store the source file's line number in a dedicated column (e.g., "line_number") within the golden table. Is this possible to achieve natively, or does it require a custom workaround?
Thanks for your feedback,
Jean-Christophe
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
https://www.semarchy.com/doc/semarchy-xdm/xdm/latest/Design/applications/action-sets.html#_data_size_limitations
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
Alexia SIMOND
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
https://www.semarchy.com/doc/semarchy-xdm/xdm/latest/Design/applications/action-sets.html#_data_size_limitations
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
-
Recover Deleted(soft Delete) Record and Configure in Application
-
How Can I Trigger Enricher or Sql Procedure when deleting?
-
Unstructured and Semi Structured Data in Semarchy?
-
Read CSV files from AWS S3
-
Find SQL Statements Generated for Match Rules
-
Is it possible to disable specific steps of an Integration Job?
-
Reset the Override on Many Records Through the Interface or By Script
-
PL/SQL Procedure With B_CREDATE and B_UPDDATE Not Mapped or Loaded, Columns Will Be Null
-
Unable to Load Publisher Data Using Continuous Load
-
LoadDataRecord contains non persitable expressions when trying to publish Data using the Rest API
See all 61 topics