Tải bản đầy đủ - 0 (trang)
11-5. Performing Delta Data Upserts with Other SQL Databases

11-5. Performing Delta Data Upserts with Other SQL Databases

Tải bản đầy đủ - 0trang

Chapter 11 ■ Delta Data ManageMent



3.



Create a new SSIS package and add two OLEDB connection managers: one (named

CarSales_Staging_OLEDB) for your destination SQL Server database (CarSales_Staging)

and one (named Source_Oracle_OLEDB) using the Oracle Provider for OLEDB to

connect to your Oracle instance. Set RetainSameConnection as True for both.



4.



Add a Data Flow task. Name it Isolate new IDs for Insert. Double-click to edit.



5.



Add an OLEDB data source. Name it Oracle Source. Edit and configure as follows:

OLEDB Connection Manager:



Source_Oracle_OLEDB



Data Access Mode:



SQL Command



SQL Command:



SELECT



EMPNO



FROM



SCOTT.EMP



6.



Click OK to confirm.



7.



Add a Lookup transform and configure to select the EMPNO from the destination

table Oracle_EMP. Join on the EMPNO column. Redirect rows that do not match to the

NoMatch output, and click OK.



8.



Add an OLEDB destination. Name it Oracle Delta and configure it as follows:

OLEDB Connection Manager:



Source_Oracle_OLEDB



Data Access Mode:



Table or View



Table or View Name:



Scott.DELTA_DATA



Your Data Flow pane should look like Figure 11-12.



Figure 11-12. Oracle delta data processing



www.it-ebooks.info



651



Chapter 11 ■ Delta Data Management



9.

10.



Return to the Control Flow pane and add a new Data Flow task. Name it Insert new

records. Double-click to edit.

Add an OLEDB source. Name it Delta Records and configure as follows

(C:\SQL2012DIRecipes\CH11\11-5_DeltaRecords.Sql):

OLEDB Connection Manager:



Source_Oracle_OLEDB



Data Access Mode:



SQL Command



SQL Command Text:



SELECT DISTINCT

S.EMPNO, S.ENAME, S.JOB, S.MGR,

S.HIREDATE, S.SAL, S.COMM,

S.DEPTNO, S.LASTUPDATED

FROM



SCOTT.DELTA_DATA D, SCOTT.EMP S



WHERE



D.EMPNO = S.EMPNO



11.



Click OK to confirm.



12.



Add an OLEDB destination. Name it Delta Output. Connect the “Delta Records”

OLEDB source to it. Map all the columns and click OK. Return to the Control Flow pane.



13.



Add an Execute SQL task and name it Delete temp table. Connect the “Insert new

records” Data Flow task to it and configure as follows:



14.



Connection:



Source_Oracle_OLEDB



SQL Statement:



DELETE FROM SCOTT.DELTA_DATA



Click OK to confirm. The Control Flow pane should look like Figure 11-13.



Figure 11-13.  Oracle SQL-based delta data processing



652

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



How It Works

Most of the current commercially available and open-source SQL databases use temporary tables in one way or

another. As it would not be feasible to show how they all can be used in this context, I have briefly outlined how

they can be used this way with Oracle, as this product is the one that SQL Server DBAs and developers tend to

encounter rather often. I can only hope that the techniques described next will be of use as a starting point if you

are using another RDBMS. Remember, if all else fails, you can always use a “normal” persisted table if you can

convince the source system DBA to allow it.

As this is essentially similar to the process described in Recipe 11-4, I have only given a brief overview, and

drawn your attention to any pertinent differences. Specifically, I have only looked at data inserts, and am leaving

you to extrapolate how to perform deletes and updates based this approach.

This technique is most appropriate when you wish to get delta data back from an Oracle data source, and

have convinced the Oracle DBA to create a global temporary table and give the account that you will use to

extract data and the rights to access the table in question. It is also fundamental to have the relevant Oracle client

software installed and configured on your server.



Hints, Tips, and Traps





Here, too, it is essential to set the Oracle OLEDB connection manager

RetainSameConnection property to True.







Equally fundamental is the “on commit preserve rows” when creating the temporary table

in the Oracle source database—without this, the process will not work.







Note that an Oracle temporary table is not like an SQL Server temp table—it physically

exists on disk. However, only the user connected during the SSIS session will be able to

use the data that they are inserting into the table. The downside to this is that the data

must be deleted afterward, or it will remain in the table and cause errors the next time

this process is run.



11-6. Handling Data Changes Without Writing to the

Source Server

Problem

You want to transfer only changed records from a source database, but you are not allowed to create ETL

metadata tables on the source.



Solution

Isolate the IDs of new or changed records, and return them as part of the SELECT query to the source database.

If your deltas are very small, the following steps explain how it can be done.

1.



On the destination database (CarSales_Staging) create a table type to hold IDs:

CREATE TYPE DeltaIDs AS TABLE (ID INT);

GO



653

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



2.



Also on the destination database (CarSales_Staging), create a stored procedure to

return the data that corresponds to the set of IDs that will be passed in using a tablevalued parameter (C:\SQL2012DIRecipes\CH11\pr_SelectDeltaData.Sql):

CREATE PROCEDURE CarSales_Staging.dbo.pr_SelectDeltaData

(

@DeltaSet DeltaIDs READONLY

)

AS

SELECT

S.ID

,S.InvoiceID

,S.StockID

,S.SalePrice

,CAST(S.HashData AS VARCHAR(50)) AS HashData

FROM



dbo.Invoice_Lines S

INNER JOIN

@DeltaSet D

ON S.ID = D.ID;



GO

3.



Create the Invoice_Lines destination table using the following DDL (and remember to

drop any previous versions if they exist). The DDL is in

C:\SQL2012DIRecipes\CH11\tbl11-5_Invoice_Lines.Sql:

CREATE TABLE dbo.Invoice_Lines

(

ID INT NOT NULL,

InvoiceID INT NULL,

StockID INT NULL,

SalePrice NUMERIC(18, 2) NULL,

HashData VARCHAR(50) NULL

);

GO



4.



Create a new SSIS package and add two OLEDB connection managers named CarSales_

OLEDB and CarSales_Staging_OLEDB, which you configure to connect to the source

and destination servers, respectively. Then add an ADO.NET connection manager

named CarSales_ADONET, which you configure to connect to the source server. This

last connection manager will be used to pass back the Table Valued Parameter (TVP)

containing the required IDs and then call the stored procedure to return delta data.



5.



Add a new package-scoped variable named InsertDeltas, which must be an object type.



6.



Add a new Data Flow task named Get Source delta information. Double-click to edit.



7.



Add an OLEDB source adapter and name it Source delta. Configure it to use the

CarSales_OLEDB connection manager. Select the ID and delta detection rows from

the source table (Invoice_Lines) using an SQL command and the following code:

SELECT

FROM



ID, Hashdata

dbo.Invoice_Lines



654

www.it-ebooks.info



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

11-5. Performing Delta Data Upserts with Other SQL Databases

Tải bản đầy đủ ngay(0 tr)

×