Data Pipeline Documentation for Nichols Concrete

Date Pipelines

The pipelines created run through the BDW custom data ETL tool. The tool is installed in the Program Files / Program Files (x86) folders respectively. Configuration and logs can be found within the tools installation directory. The configuration file holds settings related to each data source, destination, and individual pipeline jobs.

Sage Service Management (SM) Module Pipeline

The Sage Service Management pipelines are designed to provide data for the EQBI report. The data in this pipeline that is available from Service Management is: Invoices, Invoice Line Items, Work Orders, Centers, Employees, Products, and Customers. This pipeline extracts data through the Timberline Pervasive ODBC driver.

AR Module Pipeline

This pipeline pulls data from Sage AR. The only data product produced by this pipeline is AR Activity, which is used in the Aging pipeline to produce Aging data.

Aging Pipeline

The Aging pipeline differs significantly from the others. This pipeline runs on the fintech server, which also hosts the other data products and data gateway. The pipeline pulls from AR Activity data and produces the Aging data.

All pipelines are set to refresh nightly.

Snowflake Data

There are several snowflake data pipelines configured which push data to Snowflake. These pipelines use SnowSQL to ingest the raw CSV downloads (produced by Dump Truck on the Nichols production db server) and push that data into their respective tables in Snowflake.

Power BI Gateway

The MS on-premise data gataway is installed on the same server as the Aging pipeline and hosting server for the data products. This gateway is connected to the Power BI workspace. All reports/dashboards are set to refresh daily.