Data Ingest Help (DIE)

Modified on Thu, 12 Oct, 2023 at 3:57 PM

General

DIE currently supports the following file types: CSV, JSON, PSV AND XLSX.

DIE Applications

ApplicationDescriptionPermissions
ImportRun a DIE import.SF1 Super Admin, Full Access and Import Access
ConfigureConfigure mappings and logic for DIE.SF1 Super Admin, Full Access and Configure Access
LogsView logs of DIE imports.SF1 Super Admin, Full Access and Import Access
IngestsView active DIE imports.SF1 Super Admin
ApplicationsManage applications.SF1 Super Admin
UsersManage users.SF1 Super Admin
Box.comBox.com webhooks and notifications.SF1 Super Admin
HelpDIE help documentation.SF1 Super Admin, Full Access, Configure Access and Import Access

Application Schemas

Whenever you select an application, DIE will automatically check the integrity of the application's schema for any errors or potential issues. If any errors and/or potential issues are found, a modal containing this information will be displayed. If any errors are found they must be fixed in Quick Base before you can use this application within DIE. If any potential issues are found, it is suggested to fix these in Quick Base, however you will still be able to use the application within DIE.

Application Management

Box.com

You can link a Box.com folder to an application by clicking on the either the  or  button in the Box.com column for a specific application. You will then be presented with a modal, simply select the Box.com folder you would like to link and click on the Update button. Once you have linked a Box.com folder to an application, files in Box.com will become available to DIE and will allow you to set up automations.

Within the selected Box.com folder, DIE will look for files in "/Data Ingests/DIE/".
To unlink a Box.com folder from an application, select the empty value as the folder and click on the Update button.
A Box.com folder can only be linked to a single application regardless of the environment (Production or Staging).

Automations

Automations allow you to configure automatic DIE imports. When a file (data feed item) gets added to Box.com, DIE will compare automations to see if the data feed item matches any automation file values. If a data feed item matches multiple automation file values, only the first automation file value that matches is used and only one import is performed. The order in which the data feed item is compated to automation file values is as follows:

  1. Exact file match (Data Feed.json)
  2. Partial file match (Data *.json)
  3. File extension match (*.json)
  4. Any file (*)

To manage application specific automations, click on the  Automations button for an application. On this screen you will be able to view existing automations as well as add, edit and delete automations. An automation consists of two parts, the DIE configuration to use as well as the file. Some examples file values are: "Example Data Feed.xlsx", "Example*.xlsx", "*.xlsx" and "*".

Files are case-sensitive.
In order to use automations, a Box.com folder must be linked to the application.
In order to use automations, the Data Ingest user must have SF1 Super Admin premissions for the application.

Configurations

You can view information about DIE configurations by clicking on the  Configurations button for a specific application. You can also delete configurations, however for a configuration to be successfully deleted it must not exist in Quick Base.

You can also clone configurations from another application by clicking on the  Clone Configurations button. Then simply select the application you which to clone configurations from. Configurations with the same name will not be overwritten in the current application.

When cloning configurations, if the application shcemas are not EXACTLY the same then the cloned configurations will not work properly.

Linked Records

You can view how DIE will attempt to link records, as well as the fields it looks for in each table by clicking on the  Linked Records button for a specific application. You can also view any broken or missing linked records.

Broken Linked Records

Broken linked records are records that contain a reference to another record which does not exist. Broken linked records are seperated by table and broken references are highlighted in red. You can fix broken records by clicking on the  Fix Linked Records button, this will delete all broken records in the selected table if the table is an in between table (Linked Shots and Product Data, Linked Samples and Shots, etc.). If the selected table is not an in between table (Samples, Shots, etc.), all broken records will be updated, resetting the broken references.

Missing Linked Records

Missing linked records are records which should contain a reference to another record but do not. Missing linked records are seperated by table and missing reference fields are highlighted with a red background. You can download a CSV of all missing linked records for every table by clicking on the  Download as CSV button.

Purge

You can easily purge an application by clicking the  Purge button for a specific application. You will then be prompted with a modal which will allow you to select which tables to purge.

Ingest Management

In the ingests DIE application, you can view all active ingests as well as queued ingests. From here you can abort active ingests, remove queued ingests or process queued ingests manual.

Configuration

Mapping Configuration

The mapping configuration is comprised of two main components, field mappings and field types. Field mappings define the relationship between local fields (data feed items) and remote fields (application table fields). Field types define the type of data contained in a local fields as well as how they are formated.

Field Mappings

Field mapping relationships are organized by the remote table they belong to. Every mapping relationship includes two components: local field and remote field. The local field is constructed using any combination of data feed items, formulas and/or text. Any data feed items in the local field must be wrapped in square brackets. Any formulas in the local field must be wrapped in curly brackets. Some examples of a local field are: "[category]-[sku]", "0[sku]", "shot pending", "[model_name]_season_[Season]", "season_{date(Y-m-d)}" and "{lowercase([season])} season". The remote field consists of one or more application table fields. While one local field can be mapped to multiple remote fields, each remote field can only have one local field associated to it.

You can omit empty local fields by toggling the "Omit Empty" option, if the local field value is empty then the remote field will not be changed if the "Omit Empty" options is set to "Yes".

For every remote application table there are currently four actions you can perform. These actions are only applied against the field mappings contained within the selected table.

ActionDescription
Add MappingThis will create a new blank field mapping relationship at the top of the table.
Remove MappingThere are currently two operations which can be applied: remove all and remove blank. Remove all will remove every field mapping relationship, whereas remove blank will only remove blank field mapping relationships (this operation is only visible if blank field mapping relationships exist). Both operations require confirmation before anything is actually removed.
Sort MappingsThis will sort the field mapping relationships alphabetically using the local field.
Guess MappingsThis will attempt to guess the remote field for every local field and will remove any existing mappings (a confirmation is required only if there are existing field mapping relationships). If the guess is uncertain (not an exact match or alphanumeric case insensitive match) then a puzzle piece icon will be displayed next to the remote field to indicate it was an uncertain guess. If no guess could be made then the remote field will be left blank.

On top of the actions which can be performed, there are also currently two filters which allow you to show only field mapping relationships which match the filter's criteria.

FilterCriteria
BlankThis will only show field mapping relationships which are blank. For a field mapping relationship to be considered blank, the local field and/or remote field contains no value. These field mapping relationships are indicated by a blue background.
DuplicateThis will only show field mapping relationships which have duplicates. For a field mapping relationship to be considered a duplicate, the remote field must be mapped to two or more local fields. These field mapping relationships are indicated by a yellow background.

Field Types

All local fields which are contained in one or more field mapping relationships are available to configure in the fields tab (any local fields that do not exist in any field mapping relationships will be omitted). Any local fields which are mapped but are not in the data feed (for example when loading an existing mapping configuration) will show up at the top of the list with a red exclamation mark. You can mark a local field as required by toggling the required switch. As well as making a field required, you can also set the field type and format.

Field TypeField FormatDefault
CheckboxThe character(s) which indicates a checked value (separate multiple checked values with a comma).x
DateHow the date value is structured.m/d/Y
List*The character(s) which separate the list items.,
StringN/AN/A

* A list is considered a single data feed item value which is actually multiple values which are delimited.

Formulas

FormulaDescriptionsExample
dateReturns the current date with provided format.{date(Y-m-d)}
lowercaseReturns provided content in lowercase.{lowercase(Example_[field])}
uppercaseReturns provided content in uppercase.{uppercase(Example_[field])}
Nested formulas are currently not supported.

Date Formats

FormatDescriptionsExample Value
dDay of the month, 2 digits with leading zeros.01 to 31
DA textual representation of a day, three letters.Mon through Sun
jDay of the month without leading zeros.1 to 31
FA full textual representation of a month, such as January or March.January through December
mNumeric representation of a month, with leading zeros.01 through 12
MA short textual representation of a month, three letters.Jan through Dec
nNumeric representation of a month, without leading zeros.1 through 12
YA full numeric representation of a year, 4 digits.Examples: 1999 or 2003
yA two digit representation of a year.Examples: 99 or 03

All date formats are case sensitive.

Logic Configuration

The logic configuration is comprised of two different type of tasks: "Default" and "Conditional". Default tasks contain only actions which will always be performed on every data feed item on import. Conditional tasks contain local and remote conditions on top of actions. These actions are only performed on the data feed item if both the local and remote conditions are met.

Task Actions

Task actions are comprised of two parts: the operation and the remote application table to apply the operation to. For default tasks, the only operation available is insert.

OperationDescription
InsertCreate a new record.
UpdateUpdate an existing record.
Insert or UpdateUpdate an existing record, if no record exists then create one.
Only one operation can be performed per conditional task. The only exception to this is you can have multiple insert operations within one conditional task.
If no remote conditions are set, then only the insert operation can be used.
If the task action operation is either an "Update" or "Insert or Update", then the remote application table for the record being updated or inserted must have at least one remote condition applied to it. Other remote application tables may also have remote conditions applied to them in the same conditional task.
If the task action operation is an update, then the remote conditions being applied against the remote application table must return a single record.
If the task action is an "Insert or Update", either one or no records must be returned by the remote conditions applied against said remote application table.

Task Conditional Behaviors

Both local and remote conditions have a behavior, there are currently two types of behaviors:

BehaviorDescription
AllAll conditions must be true in order for the conditions to be considered as met.
AnyOnly one condition must be true in order for the conditions to be considered as met.

Local Conditions

Local conditions are made up of three parts: local field, operator and values. The local field is compared against the values specified, which are comma separated, using the defined operator.

OperatorComparison
IsOne or more specified values are equal to the data feed item value.
Is NotNo specified values are equal to the data feed item value.
Is BlankData feed item value is blank.
Is Not BlankData feed item value is not blank.
ContainsOne or more specified values exist somewhere within the data feed item value.
Does Not ContainNo specified values exist anywhere within the data feed item value.
Starts WithOne or more specified values are at the beginning of the data feed item value.
Ends WithOne or more specified values are at the end of the data feed item value.

Remote Conditions

Remote conditions consist of four components: remote application table, remote field, operator and local field. The local field is constructed using any combination of data feed items and/or text (same as the local field in the field mapping relationship).

OperatorComparison
IsLocal field value is equal to the value in the remote table field.
Is Not*Local field value is not equal to the value in the remote table field.
Is BlankField in the remote table is blank.
Is Not Blank*Field in the remote table is not blank.
ContainsLocal field value is present in the value in the remote table field (string comparison).
Does Not Contain*Local field value is not present in the value in the remote table field (string comparison).

* Negative operator.

All remote conditions are checked against the remote application at its current state before the import is run.

Remote conditions are grouped together by the remote application table. If the remote condition behavior is "All" the conditions are joined together using "and". If the remote condition behavior is "Any" then the conditions are joined together using "or".

For a remote condition to be considered as met, records must be returned from the remote application table which the remote conditions were applied against. There are two exceptions to this rule:

  1. If the task action is an "Insert or Update" and no records are returned, then that remote condition will be considered met.
  2. If a remote condition contains a negative operator ("Is Not", "Is Not Blank" and "Does Not Contain"), then the condition will only be considered met if no records in the remote application table match the condition. For example, "color is not black", will be converted to "color is black" and if no records are returned, then the condition is considered met.

Loading/Saving Configurations

Mapping and logic configurations are loaded and saved independent of each other. Meaning you can use the same mapping configuration across multiple logic configurations and vice versa. When saving either the mapping or logic configuration, the configuration name is compared against existing configurations, if a match is found, then the configuration is updated, otherwise a new configuration is created.

When loading a configuration which was saved previous to version 1.5, the configuration will automatically be converted however, you will still need to update the configuration for the changes to take effect. You will also be prompted with any changes that must be done to the configuration (if applicable).

Import

The import is divided into three stages: configuration, preparation and the actual import. During the preparation and import stage, you can abort the import by clicking on the  Abort Import button which requires confirmation.

Once you trigger an abort during either the preparation or import stage, the current packet will be completed in it's entirety before the preparation/import is truly aborted.
As of DIE 2.0, after the import preparation has finished and you choose not to run the import, the import still needs to be aborted.

Configuration

You must set the configuration which will dictate the field mappings and logic to use for the import (if you attempt to use a configuration that is pre version 1.5 you will be prompted to update the configuration before the import can be run). You can view a summary of the selected configuration by clicking on the  Configuration Summary button. You can change the default options by clicking on the Advanced Options button. Once you are ready to proceed, click on the  Create Import button to trigger the preparation of the import.

OptionDescriptionDefault
LimitThe amount of data feed items to import, set to 0 for all.0
AutomaticWhether or not to automatically start the import after preperation has completed.Disabled
Web HooksWhether or not web hooks are enabled (if enabled and web hooks fail, the import will be aborted).Disabled
XertigoWhether or not Xertigo is enabled (if enabled and Xertigo fails, the import will be aborted).Disabled
Xertigo StagingWhether or not to use the Xertigo staging server.Disabled

Prepare Import

The import preparation will determine what records will be inserted or updated in the remote application. Once complete, a list of every record that will be either inserted or updated will be shown, along with any new field choices that will be created (if applicable). On top of this, a quick reference summary of details is displayed along with information pertaining to the import (in the panel heading). Once satisfied with the import preparations, click on  Begin Import to trigger the actual import itself.

Run Import

The import will process packet items concurrently. Once a packet has been processed, linked records for every packet item will be generated and compared against current linked records to prevent both duplicate linked records and overriding existing link references.

Table RecordsRelationshipAffected Table
Shots and Product DataMany to ManyLinked Shots and Product Data
Samples and ShotsMany to ManyLinked Samples and Shots
Samples and Product DataOne to ManySamples
Shots*One to Many (Recursive)Shots

* Requires more than one record.

User Management

Currently you can view all ShotFlow1 users as well as there permissions and applications.

Logs

Every DIE import that is run is automatically logged. To view them, go to the logs section and select an application. You will then be presented with a summary of all the imports that have run. For more details, simply click on a specific record.

A log for an import is only generated once the import it is either completed or aborted.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article