Tải bản đầy đủ - 0 (trang)
11-6. Handling Data Changes Without Writing to the Source Server

11-6. Handling Data Changes Without Writing to the Source Server

Tải bản đầy đủ - 0trang

Chapter 11 ■ Delta Data Management



2.



Also on the destination database (CarSales_Staging), create a stored procedure to

return the data that corresponds to the set of IDs that will be passed in using a tablevalued parameter (C:\SQL2012DIRecipes\CH11\pr_SelectDeltaData.Sql):

CREATE PROCEDURE CarSales_Staging.dbo.pr_SelectDeltaData

(

@DeltaSet DeltaIDs READONLY

)

AS

SELECT

S.ID

,S.InvoiceID

,S.StockID

,S.SalePrice

,CAST(S.HashData AS VARCHAR(50)) AS HashData

FROM



dbo.Invoice_Lines S

INNER JOIN

@DeltaSet D

ON S.ID = D.ID;



GO

3.



Create the Invoice_Lines destination table using the following DDL (and remember to

drop any previous versions if they exist). The DDL is in

C:\SQL2012DIRecipes\CH11\tbl11-5_Invoice_Lines.Sql:

CREATE TABLE dbo.Invoice_Lines

(

ID INT NOT NULL,

InvoiceID INT NULL,

StockID INT NULL,

SalePrice NUMERIC(18, 2) NULL,

HashData VARCHAR(50) NULL

);

GO



4.



Create a new SSIS package and add two OLEDB connection managers named CarSales_

OLEDB and CarSales_Staging_OLEDB, which you configure to connect to the source

and destination servers, respectively. Then add an ADO.NET connection manager

named CarSales_ADONET, which you configure to connect to the source server. This

last connection manager will be used to pass back the Table Valued Parameter (TVP)

containing the required IDs and then call the stored procedure to return delta data.



5.



Add a new package-scoped variable named InsertDeltas, which must be an object type.



6.



Add a new Data Flow task named Get Source delta information. Double-click to edit.



7.



Add an OLEDB source adapter and name it Source delta. Configure it to use the

CarSales_OLEDB connection manager. Select the ID and delta detection rows from

the source table (Invoice_Lines) using an SQL command and the following code:

SELECT

FROM



ID, Hashdata

dbo.Invoice_Lines



654

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



8.



Add a Lookup transform, as described in steps 6 to 11 in Recipe 11-1. Here, however,

you are using the Hashdata column instead of the VersionStamp column from the

Available Lookup Columns. Name the destination column in the Lookup transform

HashData_Destination.



9.



Add a Recordset destination to the Data Flow pane. Connect the Lookup transform

to this, ensuring that you select the No Match output. Double-click to edit. Select

InsertDeltas as the variable name (to hold the output). The dialog box should look

like Figure 11-14.



Figure 11-14.  Recordset destination configuration

10.



Click Input Columns and select the ID column. This will pass only new (unmatched)

IDs through into the Recordset variable. Click OK to return to the Control Flow tab.



11.



Add a new Data Flow task and connect the preceding “Get Source delta information”

Data Flow task to it. Name it Inserts. Double-click to edit.



12.



Add a script component to the Data Flow pane, naming it Source Data. Select Source

as the Script Component type. Double-click to edit. Set the read-only variables as

InsertDeltas and the script language as Microsoft Visual Basic 2010.



655

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



13.



Click Inputs and Outputs. Expand Output 0 and rename it MainOutput. Then add five

columns, as follows:

ID



4-byte signed integer



InvoiceID



4-byte signed integer



StockID



4- byte signed integer



SalePrice



Decimal, scale 2



HashData



String, length 50



14.



Click connection managers on the left. Click Add. Name the new connection manager

DataBaseConnection. Select the ADO.NET connection manager CarSales_ADONET.



15.



Click Script on the left and then Edit Script.



16.



Add the following to the Imports region:

Imports System.Data.SqlClient



17.



Replace the ScriptMain class with the following code

(C:\SQL2012DIRecipes\CH11\TVPDeltas.vb):

Public Class ScriptMain

Inherits UserComponent

Dim connMgr As IDTSConnectionManager100

Dim sqlConn As SqlConnection

Public Overrides Sub AcquireConnections(ByVal Transaction As Object)

connMgr = Me.Connections.DataBaseConnection

sqlConn = CType(connMgr.AcquireConnection(Nothing), SqlConnection)

End Sub

Public Overrides Sub PostExecute()

MyBase.PostExecute()

End Sub

Public Overrides Sub CreateNewOutputRows()

Dim DA As New OleDb.OleDbDataAdapter

Dim TBL As New DataTable

Dim Row As DataRow = Nothing

DA.Fill(TBL, Me.Variables.InsertDeltas)

Dim cmd As New SqlClient.SqlCommand("dbo.pr_SelectDeltaData", sqlConn)

cmd.CommandType = CommandType.StoredProcedure

Dim DeltaPrm As SqlClient.SqlParameter = 

cmd.Parameters.Add("@DeltaSet", SqlDbType.Structured)

DeltaPrm.Value = TBL

Dim RDR As SqlClient.SqlDataReader = cmd.ExecuteReader()



656

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



If RDR.HasRows = True Then

Do While RDR.Read()

With MainOutputBuffer

.AddRow()

' Map data

.ID = RDR.GetSqlInt32(0)

.InvoiceID = RDR.GetSqlInt32(1)

.StockID = RDR.GetSqlInt32(2)

.SalePrice = RDR.GetDecimal(3)

If Not RDR.GetSqlString(4).IsNull Then

.HashData = RDR.GetSqlString(4)

End If

End With

Loop

End If

End Sub

End Class

18.



Click OK to confirm.



19.



Add an OLEDB destination component to the Data Flow pane. Name it Inserts.

Connect the Script source component to it. Configure it to connect to the destination

table as described in previous recipes. Click Columns and ensure that all the required

columns are selected. Click OK to confirm your changes. This makes a second trip to

the source server and adds any new records not already in the destination. The final

package will look like Figure 11-15.



Figure 11-15.  The complete package to query delta data



657

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



How It Works

There are times when session-scoped temporary tables cannot be used on a source database. In most cases, this

is because the DBA looking after the source database will not give the account that SSIS is using the necessary

rights, and/or because the additional load that this causes for the source server is deemed unacceptable. There is

another solution that can prove useful, which is to isolate the IDs of new or changed records, and return them as

part of the SELECT query to the source database. You must nonetheless convince the source system’s DBA to add a

user-defined table type and one or more stored procedures to the source system.

However, a word of caution is required. This approach is designed to handle extremely small deltas. It is

definitely not scalable to large data sets, and can cause extreme memory pressure on the destination server.

Also, this technique will only work when using SQL Server 2008 and above, as user-defined table types are only

available from this version onward.

As the two techniques shown in this recipe are extensions of the approach given in the two previous recipes,

I have not explained the entire process, but only the part that allows you to select insert delta data. The rest of the

process is nearly identical to that described earlier, and so it will not be repeated here. In these examples, I use

hashes to isolate deltas; but of course, you may prefer to use ROWVERSION or date fields.

This recipe does is the following:

Detects the delta data. However, unlike in previous recipes, the IDs to INSERT or

UPDATE will be stored in an SSIS object using a Recordset destination component.

Returns the SSIS Recordset to the source server by passing it as a table-valued

parameter (TVP) to a stored procedure in the source database.

On a more technical level, what happens is this: first, the SSIS variable object is used to populate an ADO.

NET data table. Then, this ADO.NET data table is passed back as the table-valued parameter to the stored

procedure that uses the TVP to extract a subset of data. Finally, the data subset is passed to the Script source

output buffer. What is so elegant about this approach is that an ADO.NET data table can be passed as a tablevalued parameter to an SQL Server stored procedure, with no extra processing, loops, or other work.



Hints, Tips, and Traps





Remember to add null handling for all nullable columns.







System.Data.SqlClient must be referenced in the Imports region, as it will be used to

call the stored procedure on the source database.



11-7. Detecting Data Changes with Limited Source

Database Access

Problem

You need to detect delta data when no permissions are given on the source database, except the right to read the

source table(s).



Solution

Detect the delta identifiers at the destination and then pass them back as part of the SELECT clause for the data

extraction. The following explains how to go about using this approach.



658

www.it-ebooks.info



Chapter 11 ■ Delta Data Management



1.



Create a new SSIS package. Add two OLEDB connection managers named CarSales_

Staging_OLEDB (which connects to the CarSales_Staging database) and CarSales_

OLEDB (which connects to the CarSales database). Set the RetainSameConnection

property to True for the CarSales_Staging_OLEDB connection manager.



2.



Add the following variables:



Variable Name



Type



Value



UpdateCounter



INT16



InsertCounter



INT16



DeleteTable



String



##TMP_Deletes



InsertTable



String



##TMP_Inserts



UpdateTable



String



##TMP_Updates



DeltaInserts



Object



DeltaUpdates



Object



InsertSQL



String



SELECT



ID, InvoiceID, StockID,

SalePrice, HashData



UpdateSQL



String



FROM



dbo.Invoice_Lines



SELECT



SRC.ID, SRC.InvoiceID,

SRC.StockID, SRC.SalePrice,

SRC.HashData



FROM

UpdateDataTable

3.



String



dbo.Invoice_Lines SRC



##Invoice_Lines_Updates



Add an Execute SQL task. Name it Create Temp tables on Destination. Set the

Connection to CarSales_Staging_OLEDB. Set the SQL Statement to the following

(C:\SQL2012DIRecipes\CH11\CreateTempTablesOnDestination.Sql):

IF OBJECT_ID('TempDB..##TMP_DELETES') IS NULL

CREATE TABLE TempDB..##TMP_DELETES (ID INT);

IF OBJECT_ID('TempDB..##Invoice_Lines_Updates') IS NULL

CREATE TABLE ##Invoice_Lines_Updates

(

ID INT NOT NULL,

InvoiceID INT NULL,

StockID INT NULL,

SalePrice NUMERIC(18, 2) NULL,

VersionStamp VARBINARY(8) NULL,

HashData VARCHAR(50) NULL

);



659

www.it-ebooks.info



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

11-6. Handling Data Changes Without Writing to the Source Server

Tải bản đầy đủ ngay(0 tr)

×