Data Flows

The 12d Synergy application sometimes needs to interact and exchange data with other external applications such as BIM 360, Projectwise, or with another instance of 12d Synergy application.

You need to have a Data Flows license to be able to use Data Flows.

Data flows must be configured by you, the 12d Synergy System Administrator, or a User granted with suitable permissions, with a degree of caution and good judgment, whilst keeping the business protection in mind. Simply put, the data flow will read or write from/to anywhere it has been configured to point. This could include sensitive files and/or directories, if configured in such a way. You are ultimately responsible for ensuring data integrity and security. 12d Synergy takes no responsibility for poorly configured Data Flows. If you need further clarification or guidance, please reach out to our support team for assistance.

Data flows are not compatible with 12d Model files and folders.

The Data Flows feature enables data exchange in either direction. A Data Flow Type must be configured for all applications that are communicating with 12d Synergy. A Data Flow Type includes the following two Data Flows.

1. Ingress – A data flow that is created to import data from an external source to a 12d Synergy folder. This folder in 12d Synergy is referred to as an Ingress folder. You cannot add or delete any files in the Ingress folder.

2. Egress – A data flow that is created to export data from a 12d Synergy folder to an external source. This folder in 12d Synergy is referred to as an Egress folder.

You can configure only one Ingress data flow on a managed folder or a folder. However, the managed folder must be the top-level folder of the data flow structure.

An Egress folder, on the other hand, can have multiple Egress data flows configured. This permits sharing data from 12d Synergy to multiple external applications. A single ingress and multiple egresses on the same folder are also permitted. You can also configure an egress on a managed folder.

A data flow can be enabled or disabled at two levels.

• Global level, using the 12d Synergy Administration application.

• Local Folder level, using the 12d Synergy Client application.

The data flow needs to be configured to allow for the differences between the following.

1. Naming in 12d Synergy and the external application

2. Pathing in 12d Synergy and the external application

3. How each system handles metadata and attributes

Configuring data flows has two parts.

1. Adding and configuring a Data Flow Type from the 12d Synergy Administration application.

2. Setting up either an Ingress or Egress for a Data Flow Type on a folder from the 12d Synergy Client application.

1.1. How to Add and Configure a Data Flow Type

1. You can add and configure any of the following Data Flow Types:

   • Projectwise Data Flow Type

   • BIM360 Data Flow Type

   • 12d Synergy Data Flow Type

You can add multiple Data Flow Types for an external application if there are multiple companies using the same application for content management at their end.    

1.2. How to Set up an Egress or Ingress

After you add a data flow type in the 12d Synergy Administration application and configure its basic details, you have to create an Ingress or Egress data flow from the 12d Synergy Client application.

How are deleted files handled in ingress and egress data flows?

If original copies of files are deleted in 12d Synergy, there will be no effect on the previously egressed files in the external application. However, only for the 12d Synergy egress data flow, you have the option of including the files deleted before the egress is run.

When it comes to ingress data flows, if a file is deleted after an ingress, there is an option to ingress a deleted file in the subsequent ingress that is run. This setting can be made using the Removed Source Action list when defining the ingress data flow.
However, in both the above instances, the View menu option > Show Deleted Items check box has to be selected to view any deleted files.

1.2.1. How to Create and Configure an Egress Data Flow

Egressing a file means sharing a file from 12d Synergy to a third-party tool, such as Projectwise or BIM 360.

1. Create a folder and name it Egress or something similar.

2. Edit the Egress folder and select the Data Flows tab in the Edit a Folder: <folder name>.

   

3. Click the Add button.

   The Add New Data Flow Configuration window is displayed.

   

4. Select the required data flow type that you want to use for egressing files from the Data Flow Type list.

   The data flow types configured in the 12d Synergy Administration application are displayed as options in the Data Flow Type list.

5. Select the Egress option for Direction.

6. Do one of the following:

   • Profile – If you want to configure the data flow using a profile, select this option and select the required profile from the Name list.

   • Custom – If you want to configure all the details of the data flow explicitly, select this option and enter a name for the data flow you are creating in the Name box.  

7. Enter a description of the Egress you are creating in the Description box.

8. Click the Add button.

   The created Egress is added to the left panel in the Data Flows tab.

9. Select the newly added Egress from the left panel.

   

   The Config, Schedule and Logs tabs are displayed on the right side of the Data Flows tab. By default, the data flow you are configuring is active as the Active check box is selected.

10. In the Config tab, do the following:

    1. You can choose the user who is entitled to manually run this data flow by clicking the Change User button and selecting the required user. Ensure that the selected user has the necessary read permissions to be able to run the egress.

    2. In the External path box, specify the path to the folder on the external application to which you want to egress (send out) data. You can also add an expression using the attributes of any entity mentioned in the table below to dynamically create a folder for egressing the files.

       The format of the Expression has to be - {Entity.Entity Detail}, where the Entity and Entity Details are listed in the table below.

       An example of an Expression is {User.FirstName}.

       An example of an external path with an expression - {Folder.Name}\Project Files\{User.Name}.

       • The External path box is the only mandatory box that needs to be filled while configuring a data flow. All the other fields are optional.

       • If you mention only the entity as an expression such as {Folder} or {Job}, then the folder's name or job's name is considered.  

       • If you mention only the entity as an expression, such as {User} or {Contact}, then the full name of the user or contact is considered.

       Entity	Entity Details / Attributes	Description (What is added to the path)
       {Folder}	CheckOut CreatedDate Name ParentFolder (folder) Job (job) CreatedBy	A new folder named after the folder's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.
       {Job}	ID Parent (job) Name CreatedDate (date/time) CreatedBy (contact) Description	A new folder named after the job's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.
       {User}	FirstName LastName Email Name Initials	A new folder named after the user's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.
       {Contact}	FirstName LastName Email Name Initials	A new folder named after the contact's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.

       While pointing to a location in an external application, ensure that there are no sensitive files or directories involved in the configured data flow. You are ultimately responsible for ensuring data integrity and security. 12d Synergy takes no responsibility for poorly configured Data Flows. If you need further clarification or guidance, please reach out to our support team for assistance.

       While mentioning the External path details when creating a profile, ensure you use expressions to create folders so that the Ingress or Egress folders are created dynamically for separate jobs or folders.  

       Sample entries in the External Path box while Egressing files in Projectwise and BIM360 applications

       

       
       

       

    3. If you want to name the egressed files in any specific format, refer to the Job and Folder entities and their attributes in the table above and enter an expression for the file name in the File name format (optional) box.

       • You can use only the Job and Folder entities (listed in the above table) along with their attributes for naming a file.

       • If you egress files, change the file name format and perform an egress again, then the files are duplicated but with new names that now match the new file name format.

    4. If you want to save any external references in a sub folder, then select the Save references to a sub folder check box. When this check box is not selected, the external references are saved in the same folder as the egressed files.

       An external reference is when you have a file that references data that is not within the same folder as the file.

    5. In the Attribute Mappings tab, you can map the file attributes in 12d Synergy to the file attributes used in an external application such as Projectwise.

       

       Do the following:

       • Click the button to search and select the required 12d Synergy file attribute.

         The selected attribute is displayed in the 12d Synergy Attribute column.

       • Double-click the corresponding cell in the External Attribute column and enter the name of the corresponding file attribute from the external application.

         In the image above, Approved By is a file attribute in 12d Synergy, and it is mapped to a similar file attribute named Approver in the external application.
         

         When mapping attributes, please be aware of the limitations of your chosen external application, as not all systems can handle the same metadata types as 12d Synergy.

         Mapped attributes can have display names that are different from the actual attribute name. For example, an attribute called U_CAD_Type_SewerDrainage may have a display name of CAD Type Sewer Drainage in the user interface. Please contact your external CDE Administrator to get the actual attribute name for mapping, as using the display name will not work.

       • In the Custom Values tab, click the button to add an attribute from the external application such as Approver and its default value in the corresponding cell of the External Attribute Value column. In the example below, Rob Marker is the approver.
         
         

         Whenever the added attribute is mapped, the default value is overwritten by the newly mapped value.

11. In the Schedule tab, do the following:

    

    The time at which the last egress data flow was run is displayed against the Last run label.

    1. Enter the time interval in minutes after which the egress data flow has to be run in the Frequency box. You must enter a value greater than zero in this field.  

    2. Enter the number of days after which the egress log has to be cleared in the Log autoclear duration box.

12. In the Logs tab, you can configure the criteria for generating egress log files.

    

    • Select the start and end dates of the period for which you want to generate the logs from the Start and End list.

    • If you want the logs to include details of the errors encountered while egressing files, select the Errors check box.

    • If you want the logs to include the warnings encountered while egressing files, select the Warnings check box.

    • If you want the logs to include general details such as the beginning and completion of a data flow while egressing files, select the General check box.

    • If you want the logs to include the debugging details while egressing files, select the Debug check box.

      • Click the Export button to create a log file for a selected period and export it to a location in the 12d Synergy Client application.  

      • Click the Delete Logs button to delete the logs created only for this data flow.

      • Click the Truncate Logs button to permanently delete the logs created for all the data flows.

      • Click the Show button to generate the log based on the configuration. The details are displayed in the table.

      • If you double-click any log entry in the table to see all details of the log entry in the <Dataflow name> Log Line View window.

        

      • You can delete a data flow by simply selecting it from the left panel of the Data Flows tab and clicking the Remove button.

13. Click the Save button.

    The Egress data flow is configured.

1.2.2. How to Create and Configure an Ingress Data Flow

Ingressing a file means importing it from an external application, such as Projectwise, into 12d Synergy.

1. Create a folder or a managed folder and name it Ingress folder.

2. Edit the Ingress folder and select the Data Flows tab in the Edit a Folder: <folder name>.

   

3. Click the Add button.

   The Add New Data Flow Configuration window is displayed.

   

4. Select the required data flow that you want to use for egressing files from the Data Flow Type list.

   The data flow types configured in the 12d Synergy Administration application are displayed as options in the Data Flow Type list.

5. Select the Ingress option for Direction.

6. Do one of the following:

   • Profile – If you want to configure the data flow using a profile, select this option and select the required profile from the Name list.

   • Custom – If you want to configure all the details of the data flow explicitly, select this option and enter a name for the data flow you are creating in the Name box.  

7. Enter a description of the data flow you are creating in the Description box.

8. Click the Add button.

   The created data flow is added to the left panel in the Data Flows tab.

9. Select the newly added data flow from the left panel.

   

   The Config, Schedule and Logs tabs are displayed on the right side of the Data Flows tab. By default, the data flow you are configuring is active as the Active check box is selected.

10. In the Config tab, do the following:

    1. You can choose the user who is entitled to manually run this data flow by clicking the Change User button and selecting the required user. Ensure that the selected user has the necessary read permissions to run the ingress.

    2. In the External path box, specify the path to the folder on the external application from which you want to ingress (receive) data. You can also create expressions using the attributes of any entity mentioned in the table below to dynamically locate the apt folder in 12d Synergy for ingressing the files.

       The format of the Expression has to be - {Entity.Entity Detail}, where the Entity and Entity Details are listed in the table below.

       An example of an Expression is {User.FirstName}.

       An example of an external path with an expression - {Folder.Name}\Project Files\{User.Name}.

       • The External path box is the only mandatory box that needs to be filled while configuring a data flow. All the other fields are optional.

       • If you mention only the entity as an expression such as {Folder} or {Job}, then the folder's name or job's name is considered.  

       • If you mention only the entity as an expression, such as {User} or {Contact}, then the full name of the user or contact is considered.

       Entity	Entity Details	Description (What is added to the path)
       {Folder}	CheckOut CreatedDate Name ParentFolder (folder) Job (job)CreatedBy	A new folder, named after the folder's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.
       {Job}	ID Parent (job) Name CreatedDate (date/time) CreatedBy (contact) Description	A new folder, named after the job's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.
       {User}	FirstName LastName Email Name Initials	A new folder, named after the user's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.
       {Contact}	FirstName LastName Email Name Initials	A new folder, named after the contact's details mentioned in the Entity Details column of this table, is created at the location mentioned in the External path box.

       While pointing to a location in an external application, ensure that there are no sensitive files or directories involved in the configured data flow. You are ultimately responsible for ensuring data integrity and security. 12d Synergy takes no responsibility for poorly configured Data Flows. If you need further clarification or guidance, please reach out to our support team.

       Sample entries in the External Path box while Ingressing files in Projectwise and BIM360 applications

       

       

    3. If you want to name the ingressed files in any specific format, refer to the Job and Folder entities and their attributes in the table above and enter an expression for the file name in the File name format (optional) box.

       • You can use only the Job and Folder entities (listed in the table above) alone with their attributes for naming a file.  

       • If you ingress files, change the file name format and perform an ingress again, then the files are duplicated but with new names that now match the new file name format.

    4. When you ingress data in to 12d Synergy, all the files that are ingressed are displayed in the Files tab of the Ingress Folder dashboard view. Consider a case where after you ingress a file from an external source, which after ingressing is deleted in the external source. When the ingress is run the next time, the deleted file can be shown as a deleted file in 12d Synergy, or it can be simply ignored and remain as a file in the Ingress folder. This setting can be made using the Removed Source Action list.

       • Select the Ignore option to let the file be as it was in the Ingress folder before its deletion in the external drive.

       • Select the Mark As Deleted option to mark  the file as deleted in the State column of the Folder dashboard view.

         

         To view the deleted files, ensure that the 12d Synergy Client application > View menu option > Show Deleted Items option is selected. A tick mark is displayed against the option when it is selected.  Only if this option is selected and the Mark As Deleted option is selected in the Removed Source Action list of the Edit a Folder: <folder name> , you will be able to view the Deleted status in the State column of the Folder dashboard view.

         

    5. In the Attribute Mappings tab, you can map the file attributes in 12d Synergy to the file attributes used in an external application such as Projectwise.

       

       Do the following:

       • Click the button to search and select the required 12d Synergy file attribute.

         The selected attribute is displayed in the 12d Synergy Attribute column.

       • Double-click the corresponding cell in the External Attribute column and enter the name of the corresponding file attribute from the external application.

         In the image above, Approved By is a file attribute in 12d Synergy, and it is mapped to a similar file attribute named Approver in the external application.  

         When mapping attributes, please be aware of the limitations of your chosen external application, as not all systems can handle the same metadata types as 12d Synergy. Mapped attributes can have display names that are different from the actual attribute name. For example, an attribute called U_CAD_Type_SewerDrainage may have a display name of CAD Type Sewer Drainage in the user interface. Please contact your external CDE Administrator to get the actual attribute name for mapping, as using the display name will not work.

11. In the Schedule tab, do the following:

    

    The time at which the last ingress data flow was run is displayed against the Last run label.

    1. Enter the time interval in minutes after which the ingress data flow has to be run in the Frequency box. You must enter a value greater than zero in this field.  

    2. Enter the number of days after which the ingress log has to be cleared in the Log autoclear duration box.

12. In the Logs tab, you can configure the criteria for generating ingress log files.

    

    • Select the start and end dates of the period for which you want to generate the logs from the Start and End list.

    • If you want the logs to include details of the errors encountered while ingressing files, select the Errors check box.

    • If you want the logs to include the warnings encountered while egressing files, select the Warnings check box.

    • If you want the logs to include general details such as the beginning and completion of a data flow while ingressing files, select the General check box.

    • If you want the logs to include the debugging details while ingressing files, select the Debug checkbox.

      • Click the Export button to create a log file for a selected period and export it to a location in the 12d Synergy Client application.  

      • Click the Delete Logs button to delete the logs created only for this data flow.

      • Click the Truncate Logs button to permanently delete the logs created for all the data flows.

      • Click the Show button to generate the log file based on the configuration.

      • If you double-click any log entry in the table, you will see all details of the log entry in the <Dataflow name> Log Line View window.

        

      • You can delete a dataflow by simply selecting it from the left panel of the Data Flows tab and clicking the Remove button.

13. Click the Save button.

    The Ingress data flow is configured.

1.3. How to Disable a Data Flow Type

1.3.1. Globally using the 12d Synergy Administration application

1. In the 12d Synergy Administration application, select the Data Flows > Data Flows tab.

   All the added data flows are displayed in the left panel.

   

2. Select the data flow type you want to disable from the left panel and click the Disable button under the left panel.

   The following confirmation message is displayed.

   

3. Click the Yes button.

   All the data flows created for the selected type across all your jobs in 12d Synergy are disabled.

1.3.2. Locally using the 12d Synergy Client application

1. In the 12d Synergy Client application, access the folder for which you want to disable a data flow type.

2. Right-click the folder and select the Edit Folder option.

   The Edit a folder: <Egress folder name> window is displayed.

3. Select the Data Flows tab.

4. Select the data flow type you want to disable from the left panel of the Data Flows tab.

   

5. Clear or deselect the Active check box in the Config tab.

6. Click the Save button.

   All the data flows created on the folder for the selected external application are disabled.    

1.4. How to Update a .dll File for a Data Flow

When you receive a new .dll file from us for an external application you are already connected to, you will need to update the existing one. It is recommended that this procedure be done after business hours, as it requires a restart of the 12d Synergy Server.

1. Save the new .dll file in a local location.

2. In the 12d Synergy Administration application, select the Data Flows > Data Flows tab.

3. Select the data flow for which the .dll needs an update.
   

4. Click the Update button.
   The Update task library window is displayed.
   

5. Click the ellipsis button against the Path box.

6. Browse and select the new .dll file from the local location and click the Upload button.
   The new .dll file is uploaded and the following confirmation message is displayed.
   

7. Click Ok.

8. To restart the 12d Synergy Server, click the Restart button in the 12d Synergy Administration application > General tab > Maintenance tab > Restart tab.
   Once the restart is done, the new .dll will take effect.

It is recommended that this procedure be performed after business hours, as it requires restarting the 12d Synergy Server.

1.5. Profiles

A profile is like a template for defining a data flow. When you configure and create a data flow that has to be replicated for different folders in the same job or in different jobs, then instead of entering all the configuration details multiple times, you can just create a profile and use that profile whenever you are creating a data flow. However, some details still have to be configured when creating the data flow.

1.5.1. How to Create and Configure a Profile

1. In the 12d Synergy Administration application, select the Data Flows > Data Flows tab.

   The added data flow types are displayed in the left panel.

   

2. Select the data flow type for which you want to create a profile from the left panel and click the Profiles tab.

   The Ingress Profiles and Egress Profiles tabs are displayed. Let us create an Ingress profile in this procedure.  

3. Click the Ingress Profiles tab.

   

4. Click the button to create an ingress profile.

   The Create Data Flow Profile window is displayed.

   

5. Enter a name and description for the data flow profile in the Name and Description boxes.

6. Click the OK button.

   The profile is created and is displayed in the left panel of the Ingress Profiles tab.

   

7. Select the newly created profile from the left panel of the Ingress Profiles tab.

   The Config and Schedule tabs are displayed in the right panel of the Ingress Profiles tab.  

8. Configure the fields displayed in the Config tab. (Please refer to step 10 of section 1.2.2.)

   While mentioning the External path details for a profile, ensure you use the expressions to create folders so that the Ingress or Egress folders are created dynamically for separate jobs or folders.  

9. Configure the fields displayed Schedule tab. (Please refer to step 11 of section 1.2.2.)

   Note that while creating profiles, there are only some fields to configure when compared to the fields that are displayed while configuring a data flow in the 12d Synergy Client application.

10. Click the Save button.

    The ingress profile is now created.

    • You can create multiple ingress or egress profiles for a data flow.

    • A profile created for one data flow type is applicable only to that data flow type. This means that if you create a profile for a BIM360 data flow type, it will NOT be applicable to the Projectwise data flow type.

    • If you want to have ingress and egress profiles for a data flow type, you must create them explicitly for that data flow type.

1.6. How to Export Data Flow Logs

1. After generating a log in the Edit a Folder window > Data Flows tab >  Logs tab for either an Egress or Ingress folder, click the Export button.

   

   The Export Settings window is displayed.

   

2. Do one of the following:

   • Select the period for which you want to generate a log using the Start and End calendar lists.

   • Select the All Time check box to generate a log from the time that data flow created.

3. In the How would you like the file to be saved  list, select one of the following options:

   • Single File – to export the report as a one off instance

   • Daily – to export day-wise report for the set period

   • Monthly – to export a month-wise report for the set period

   • Yearly – to export a year-wise report for the set period

4. Click the Export button.

   The Find a job window is displayed.

5. Browse and select the required job and folder to which you want to export the log, enter a file name for the log and click the Select button.

   

   The data flow log is created and is saved at the selected location, and a confirmation message is displayed.  

1.7. How to Manually Run a Data Flow

While the 12d Synergy Administrator can schedule when a data flow runs, an end user can also manually run it from the 12d Synergy Client application.

To run a data flow, right-click the Ingress or Egress folder and select the Data Flows > Run Ingress option (or Run Egress, Run All Egress option).  

Based on the folder you select, and the number of data flows configured on it, the right-click options change.  

1.8. How to View a Summary of Dataflows

This feature is available in 12d Synergy version 5.1.7 or later.

1. In the 12d Synergy Administration application, select the Data Flows tab > Dashboard tab. All the Ingress and Egress data flows that are created in your project (jobs) are displayed. You can view all the details of each data flow in the table.

   

2. You can do any of the following from this window.

   • Filter by the type of data flow by selecting the required option from the Filter by Type list and clicking the Search button.

   • Filter the data flows by the job in which they are created by entering the name of the required job in the Job Filter box.

   • Select the number of items you want to see on each page from the Items per page list.

   • Check the status of each data flow by seeing the State column. You can have only three states for a data flow. They are OK, Errors and Running.

   • Double-click a row or click the View/Edit link to view or/and edit the details of the data flow in the Edit a Folder: <Folder Name> window. (Refer to step 9 of  section 1.2.2.)

   • Click the Run link to run the data flows. The link is changed to Abort and you can click that link if you want to stop the data flow from running further.

   • Click the Disable link to disable the data flow. Once disabled, the link is then displayed as Enable and you can enable the data flow by clicking on the Enable link.

   • Click the Export button to export the data flows dashboard to a .csv file and save it in any chosen location in your jobs.