Tải bản đầy đủ - 0 (trang)
15-10. Creating a Centralized Logging Framework

15-10. Creating a Centralized Logging Framework

Tải bản đầy đủ - 0trang

Chapter 15 ■ Logging and Auditing



,@ErrorDescription VARCHAR(MAX) = NULL

,@ErrorLineNo INT = NULL

,@ErrorSeverity INT = NULL

,@ErrorState INT = NULL

)



AS



INSERT INTO CarSales_Logging.log.EventDetail

(

Process

,Step

,StartTime

,Comments

,ErrorNo

,ErrorDescription

,ErrorLineNo

,ErrorSeverity

,ErrorState

)



VALUES

(

@Process

,@Step

,@StartTime

,@Comments

,@ErrorNo

,@ErrorDescription

,@ErrorLineNo

,@ErrorSeverity

,@ErrorState

) ;

GO

3.



Add logging for all your T-SQL-based processes, like this:

-- Start Header ------------------------------------------------------------DECLARE @StartTime DATETIME

DECLARE @ ProcName VARCHAR(150) --- The calling procedure

DECLARE @ProcStep VARCHAR(150) -- The step in the procedure

SET @ProcName = OBJECT_NAME(@@PROCID)



BEGIN TRY

---- End Header ----------------------------------------------------------- 

SET @ProcStep = 'My test'

   

--Your code here - 



878

www.it-ebooks.info



Chapter 15 ■ Logging and Auditing



EXECUTE CarSales_Logging.log.pr_Logging_Simple @ProcName, @ProcStep, @StartTime,

'Part of the process OK'



-- Start Footer ----------------------------------------------- 

END TRY

BEGIN CATCH



IF @@trancount > 0 ROLLBACK TRAN

SET XACT_ABORT OFF;



DECLARE @Error_Number int

DECLARE @ ErrorDescription VARCHAR(MAX)

DECLARE @ErrorLine INT

DECLARE @ErrorSeverity INT

DECLARE @ErrorState INT

DECLARE @Message VARCHAR(150)



SET @ERROR_NUMBER = ERROR_NUMBER()

SET @ErrorDescription = ERROR_MESSAGE()

SET @ErrorLine = ERROR_LINE()

SET @ErrorSeverity = ERROR_SEVERITY()

SET @ErrorState = ERROR_STATE()



EXECUTE CarSales_Logging.log.pr_Logging_Simple @ProcName, @ProcStep, @StartTime,

'Error - Stored Procedure', @ERROR_NUMBER, @ErrorDescription, @ErrorLineNo,

@ErrorSeverity, @ErrorState



SET @Message = 'Error in procedure: ' + @ProcName

RAISERROR(@Message ,16,1)



END CATCH

-- End footer ------------------------------------------4.



Create an ADO.NET connection manager named CarSales_Logging_ADONET,

which connects to the CarSales_Logging database.



5.



Add Logging to SSIS tasks as follows. For a task or container whose outcome you wish

to log (in this example it will be a Data Flow task).



6.



Add the following two variables at the scope of the task (the safest way to do this is to

select the task before creating the variables):



Variable Name



Type



Value



IsError



Boolean



False



StartTime



DateTime



879

www.it-ebooks.info



Chapter 15 ■ Logging and Auditing



7.



With the task selected, click the Event Handlers tab, and select OnPreExecute from

the list of available event handlers.



8.



Click “Click here to select . . . ” and then add an Execute SQL task onto the Event

Handlers pane. Name it Get StartTime. Configure it to use the ADO.NET connection

CarSales_Logging_ADONET.



9.



Add the following parameter to the Parameter Mapping pane:



10.



Variable



Direction



Type



ParameterName



User::TaskStartTime



Output



DateTime



@TaskStartTime



Add the following SQL as the SQL Statement:

SELECT @TaskStartTime = GETDATE()



11.



Close and confirm the task. This will pass the task start time to an SSIS variable

scoped at task level.



12.



Select OnError from the list of event handlers and click “Click here to select . . . ”. Add

an Execute SQL task onto the Event Handlers pane. Name it Log Failure. Add the

ADO.NET connection to the database used for logging.



13.



For the Log Failure Execute SQL task, add the following parameters:



14.



VariableName



Direction



DataType



ParameterName



User::ErrorCode



Input



Int32



@ErrorNo



User::ErrorDescription



Input



String



@ErrorDescription



System::PackageName



Input



String



@Process



System::SourceName



Input



String



@Step



User::StartTime



Input



DateTime



@StartTime



User::IsError



Input



Boolean



@IsError



In the Log Failure SQL task, add the following SQL as the SQL Statement to execute:

SET @IsError = 1

EXECUTE dbo.pr_LogEvents@Process, @Step, @StartTime, 'SSIS Error', @ErrorNo,

@ErrorDescription



15.



Select OnPostExecute from the list of event handlers, and click “Click here to select . . . ”.

Add an Execute SQL task onto the Event Handlers pane. Name it Log Success. Add

the ADO.NET connection to the database used for logging.



880

www.it-ebooks.info



Chapter 15 ■ Logging and Auditing



16.



17.



Add the following parameters:



VariableName



Direction



DataType



ParameterName



User::TaskStartTime



Input



DateTime



@TaskStartTime



System::PackageName



Input



String



@Process



User::IsError



Input



Boolean



@IsError



In the Log Success SQL task, add the following SQL as the SQL Statement to execute:

IF @IsError = 0

BEGIN

EXECUTE dbo.pr_LogEvents @Process, @Step, @StartTime

END



18.



Return to the Control Flow pane.



How It Works

If you have built—or are building—a moderately complex SSIS package to load data, then you probably also want

to be able to log all relevant information to a central repository. Equally probably, this could be an SQL Server

table rather than a text file. The reasons for this are that





A central point of focus avoids having to search through multiple sources of information.







A T-SQL table can be searched and filtered very rapidly.



So what are the advantages and disadvantages of logging to T-SQL using “roll your own” logging rather than

using and extending the built-in solutions proposed by SSIS? Table 15-8 provides a succinct overview.

Table 15-8.  Advantages and Disadvantages of Creating a Custom Logging Framework



Advantages



Disadvantages



Extremely useful to store a centralized record of process flow—or More effort to implement.

to put it simply, a list of what worked and what did not.

Highly customizable.



Tricky to debug.



Logging highly targeted and specific information does not cause

resource contention.



Easy to forget to add a log call to tasks and

code.



Only required events and information are logged.



Can itself be a source of package errors and

cause package failure.



Easy-to-specify duration of data storage. Keeps only recent

information.

Simple to extract high-level data for baselining.

Can help in debugging.



881

www.it-ebooks.info



Chapter 15 ■ Logging and Auditing



Nevertheless, I am firmly convinced that logging to a central table is very well suited to production ETL jobs,

and I have been applying it for years. If anything, it becomes a habit and a valuable assistance, not only after

packages have gone into production, but also when debugging them.

You may want to suggest that building custom logging frameworks is a thing of the past now that SQL Server 2012

has the SSIS catalog to provide logging. My response is that while the SSIS catalog is a wonderful tool for logging SSIS

tasks and packages, it is not (at least not yet) a centralized repository for logging all events, including those generated

by stored procedures. However, if SSIS handles your entire ETL process without any calls to stored procedures, then

SSIS catalog logging is certainly the way to go. The SSIS catalog is described in Recipes 15-16 to 15-18.

The approach that I am suggesting here is to have a single SQL Server table to log event data generated when

an SSIS task or T-SQL code fragment succeeds or fails. A single stored procedure logs the outcome. This stored

procedure is called from T-SQL or from SSIS. In any case, for the reasons outlined at the start of this chapter, I also

strongly advise that any logging use its own database.

The DDL for the log table explains its use. As the terminology is different between T-SQL and SSIS, some

choices of vocabulary have been made for the table field names, but I hope that they are self-explanatory. The

only fields that might need further explanation are Step and Process. I have added these because a package can

potentially log many events, and these events are nested (stored procedures call other ‘sprocs, SSIS packages

call SSIS packages, which in turn have tasks). These two fields allow you to track which process calls what sub

element, and consequently creates a simple hierarchy to track this nesting of calls. It means always containing

the reference to the





Process: The container (the SSIS package or stored procedure).







Step: The element or task to log.



I would suggest that this approach is useful for small- to medium-sized SSIS processes, where essential—but

not over-detailed logging—is required. For very complex ETL systems, you will need to extend these tables and

procedures to suit your specific requirements.

A centralized approach like this is going to need to store three types of elements:





Data common to SSIS and T-SQL.







Data specific to either SSIS or T-SQL.







Error details (which can also be common to both SSIS and T-SQL or specific to one

or the other).



So, as the assumptions underlying this approach are that you want fairly minimal logging, let us assume that

the following basic principles need to be logged for each step:





The package process







The package step







A description of what is logged







Start time







End time (the time that the event is logged)







Error details if an error is caught



You will need the following:





A table to store success.







A stored procedure to log event outcomes.







An ADO.NET connection manager to the database used for logging. I suggest ADO.NET over

OLEDB purely for the ease with which it can handle parameter passing, especially when

there are multiple parameters. You can use an OLEDB connection manager if you wish.



882

www.it-ebooks.info



Chapter 15 ■ Logging and Auditing



Logging from T-SQL becomes a combination of straight logging (as described in Recipe 15-7) and error

handling (described in Recipe 15-8). Here is an example of a stored procedure template where I have wrapped

the error handling in a standard header and footer, and used the same stored procedure to log successful steps as

well as any failures. This stored procedure outline can even become the model for all your stored procedures in

an ETL process.

Logging from SSIS to a centralized table (the same one that you use when logging from stored procedures)

necessitates judicious use of event handlers to work cleanly and efficiently. What you can do for most tasks is use

the following:





The OnPreExecute event handler to set the time at which the task began.







The OnPostExecute event handler to log the task completion.







The OnError event handler to log the fact that the task has failed, along with any error

messages.



Also, you will need to set two task-scoped variables (the scope is important) so that the start time can be

initialized, and an error flag set (which will prevent both the OnPostExecute event handler from logging the event

as well as the OnError event handler, should an error occur). In other words, flagging the IsError variable to

True as part of the OnError event allows the error state to be picked up by the log process which occurs when the

OnPostExecute event fires. As the Log Success task also tests this variable when it fires, it will only call the stored

procedure to log an event when an error has not occurred.

The use of Getdate() while pointing at the server that performs the logging can be important, but you

cannot be sure that all your servers will be synchronized to return the exact same date and time. So if you are

using a script package and the Now() function on a server hosting SSIS, while logging takes place on another

server, you could end up with incoherent timestamps.



Hints, Tips, and Traps





If your SSIS package is so simple that it contains only one step, then you can use the

System::StartTime system variable to record the start time, and you will not need to

use—or set—the TaskStartTime variable.







Do not be tempted to use the ContainerStartTime system variable to record package

start time—this will give the start time of the Execute SQL task performing the logging—

not the task you are trying to monitor.







When logging from T-SQL, the order of the input parameters for the stored procedure is

important, as any noncompulsory (NULL) parameters must follow compulsory ones.



15-11. Logging to a Centralized Framework When Using

SSIS Containers

Problem

When using SSIS containers and a custom logging framework based on event handlers, you find that the same

events are logged many times over.



883

www.it-ebooks.info



Chapter 15 ■ Logging and auditing



Solution

Apply your custom logging framework in a specific way to SSIS containers so that only one log entry is made.

Here is how it can be done:

1.



Create an SSIS package.



2.



Create an ADO.NET connection manager named CarSales_Logging_ADONET that

connects to the CarSales_Logging database—unless it exists already, of course.



3.



Create the following variables:



4.



5.



Name



Scope



DataType



ContainerStart



Package



DateTime



Add an Execute SQL task. Name it SetStartTime. Double-click to edit and configure it

as follows:

ConnectionType:



ADO.NET



Connection:



CarSales_ADONET



SQLSourceType:



Direct Input



SQLStatement:



SELECT @ContainerStart = GETDATE()



IsQueryStoredProcedure:



False



Click ParameterMapping on the left, and add the following parameter:



VariableName



Direction



DataType



ParameterName



ContainerStart



Output



DateTime



@ContainerStart



6.



Click OK to complete your modifications



7.



Connect this Execute SQL task to the container you wish to log.



8.



Add two Execute SQL Tasks. Connect both to the container you wish to log.

Double-click one of the precedence constraints and set it to Failure. Name one

Log Success and the other (connected via the precedence constraint set to Failure)

Log Failure.



9.



For both of these tasks, add the ContainerStart parameter as described in step 3, but

with the direction set to Input.



10.



For both of these tasks, double-click to edit and set the following:

ConnectionType:



ADO.NET



Connection:



CarSales_Staging_ADONET



SQLSourceType:



Direct Input



IsQueryStoredProcedure:



False



884

www.it-ebooks.info



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

15-10. Creating a Centralized Logging Framework

Tải bản đầy đủ ngay(0 tr)

×