Replicating Data SAP Applications to BigQuery WithSAP Data Services

You can now query data directly through virtual tables by using the hyper-scaler storage to integrate data like Google BigQuery with business data in SAP Data Warehouse Cloud that is based on SAP Business Technology Platform. The advantage here is that you can provide live connectivity along with the ability to query Google BigQuery data from SAP Data Warehouse Cloud, leading to critical analysis of business data.

Some benefits to users of data replication from SAP to Big Query are analyzing SAP and third-party data in one place and maximizing ROI in current SAP and Google services. You can also get real-time analytics without data replication and ensure live connectivity to query BigQuery by virtual tables.

This post focuses on replicating SAP applications to BigQuery by using SAP Data Services.

The database can be SAP HANA or any other but with the precondition that it is supported by SAP. The data replication can be used to back up data in SAP to combine the data from SAP systems with data from other systems in BigQuery. This can be used to get insights from ML (Machine Learning) for petabyte-scale data analytics. Replication from SAP to Big Query can be done easily and seamlessly by SAP system administrators who are well-versed with the configuration of SAP Basis, SAP DS, and Google Cloud.

Prerequisites for SAP to BigQuery data replication

The methodology for SAP to Big Query assumes that the database server, SAP Data Services, and the SAP application system are preinstalled and configured for the usual operations. You should check with the SAP to make sure that the planned configuration is as per the SAP licensing requirements. However, the requirements might vary based on whether you are exporting data from an ancillary database or an SAP application system. 

Challenge of SAP to Big Query data replication

The main challenge to SAP to Big Query data replication is that critical business data in various sources have to be federated to offer business value while building analytical dashboards. This is because organizations want to integrate the data located in the hyper-scaler storage such as Google Big Query with the data residing in SAP Data Warehouse Cloud.

The solution to this issue is to connect SAP to Big Query so that data queried from Google BigQuery is available in SAP Data Warehouse Cloud without the need for replication. Once this live connectivity is established, businesses can visualize data integrated into a single source with SAP Analytics Cloud using different analytical dashboards.

The outcome is richer insights into business data once data from SAP Data Warehouse Cloud is integrated with data present in Google BigQuery services. Queries are federated through virtual tables with current and relevant data that is not replicated or cached from its source. 

SAP to Big Query data replication

The flow of SAP to Big Query data replication is smooth and seamless. SAP Data Services first extracts the data from the SAP application or an underlying database. This data is then transformed or formatted to match the architecture of the Big Query. Finally, the load job moves the data to Big Query, and once completed, the data is available in Big Query for analysis.

At the time of export, it is possible to preset when the SAP Data Service should initiate the process. All data present in the target BigQuery table is overwritten by the newly exported data. After the replication process is completed the data in BigQuery is not kept in sync with the data in the source system. 

The SAP Replication Server leverages the CDC (Change Data Capture) support of SAP Data Services that includes the provisioning of data and delta capabilities for all source tables in real-time.

Given here is the flowchart of SAP to Big Query data movement.

• Data is updated by SAP applications in the source system.
• Data changes are replicated by the SAP LT Replication Server and the data stored in the Operational Delta Queue
• A subscriber of the Operational Delta Queue, SAP DS, tracks the queue for data changes at pre-set intervals.
• SAP DS pulls the data from the delta queue, formats the data to make it compatible with the BigQuery structure, and starts the process that loads the data from SAP to Big Query.
• Data is now available in BigQuery for analysis.

This is how data moves from SAP to BigQuery.

Tools for replicating data from SAP to Big Query

While there are several tools to replicate SAP to Google BigQuery, knowing and using the best and most optimized ones makes the movement easy and without any complexities. The one critical thing to consider is whether the tool supports replication from the SAP runtime versions or access to a database is required. The best tools support both methods regardless of whether you are looking for replication from SAP runtime versions from the application layer or the database layer.

Look for these attributes while choosing an SAP to Big Query tool.

• Capable of replicating huge volumes of SAP data quickly to the BigQuery database.
• A fully automated tool that does away with the need for constant supervision of DBAs. Hence, the TCO is quite low as coding is not required. SAP data replication, SCD type history, data transformation, and data reconciliation are all done through a point and click interface. Specialized manpower is not needed to monitor SAP to Big Query data replication.
• The best tools for SAP to Big Query maintain the referential integrity of data that is, the exact date, time, and the values that change at the columnar level are on record.

Integrating data from SAP to Big Query helps to transform data optimally for better understanding and analytics.