Skip to main content

Pipeline Parallelization

One of the new features of SQL Server Integration Services 2008 is the parallelization of the pipeline. This is implemented by creating multiple execution threads that can be run on different processors, increasing the performance of the package. In SSIS 2005, this separation of threads could be emulated by using a Union All transformation to create a new buffer.

If you turn on logging in the data flow, you can see these separate execution trees in the events PipelineExecutionPlan and PipelineExecutionTrees. I created a simple package that creates a row set of integers, multicasts to a conditional split (as a terminator) and a flat file destination.

Here are the messages from those events:
User:PipelineExecutionTrees
Begin Path 0
output "Output 0" (32); component "Script Component" (29)
  input "Multicast Input 1" (53); component "Multicast" (52)
  Begin Subpath 0
    output "Multicast Output 1" (54); component "Multicast" (52)
    input "Flat File Destination Input" (64); component "Flat
      File Destination" (63)
  End Subpath 0
  Begin Subpath 1
    output "Multicast Output 2" (69); component "Multicast" (52)
    input "Conditional Split Input" (57); component
      "Conditional Split" (56)
  End Subpath 1
End Path 0


User:PipelineExecutionPlan
Begin output plan
  Begin transform plan
  End transform plan
  Begin source plan
    Call PrimeOutput on component "Script Component" (29)
      for output "Output 0" (32)
  End source plan
End output plan

Begin path plan
  Begin Path Plan 0
    Call ProcessInput on component "Multicast" (52)
      for input "Multicast Input 1" (53)
    Create new execution item for subpath 0
    Create new execution item for subpath 1
    Begin Subpath Plan 0
      Call ProcessInput on component "Flat File Destination" (63)
        for input "Flat File Destination Input" (64)
    End Subpath Plan 0
    Begin Subpath Plan 1
      Call ProcessInput on component "Conditional Split" (56)
        for input "Conditional Split Input" (57)
    End Subpath Plan 1
  End Path Plan 0
End path plan


You can see that the new log creates a subpath for both the flat file destination path and the conditional split path. The best part is that each subpath can then be executed on different processors, increasing the speed of the package!

Version: SQL Server 2008 CTP6

Comments

Popular posts from this blog

SSIS Configuration to Configuration to Configuration Schema

I've gotten several requests to put down in writing the configuration schema that I use as the base of my SQL Server Integration Services framework. It contains a set of configurations: an indirect environment variable configuration, which points to an XML configuration file configuration, which points to a SQL Server configuration. I learned this configuration from the Project REAL Reference Implementation . If you're getting started with a BI implementation, I highly recommend that you download it for some great ideas and best practices. Steps to implement: 1) Create an environment variable on your machine with the name of SSIS_CONFIG_FILE and the value of: C:\SSIS\Config\MasterConfigFile.dtsConfig. 2) Create an SSIS configuration file at C:\ SSIS\Config\MasterConfigFile.dtsConfig with the line: <configuration valuetype="String" path="\Package.Connections[CONFIG_SERVER].Properties[ConnectionString]" configuredtype="Property"><configuredv

Execute SQL Task Designer Limit

After migrating a package from DTS to SSIS, I had a problem with an Execute SQL Task. I couldn't change any characters in the SQLStatement property; I couldn't add any new characters; I could delete characters, but not retype them! After googling several variations of "integration services" "read only" and "Execute SQL Task", I deleted about half of the entry in a fit of frustration. Lo and behold, I could type again. Apparently, there is limit on the size or number of characters that can be entered in the SQLStatement property. From my experimentation, I came up with a limit of 32767 characters. The interesting thing is that the restriction only seems to be on the designer. If you set the SourceType to "Variable" and use a variable that contains more than 32767 characters, the task will execute. Also, if you use the "Direct Input" SourceType and modify the package XML to set the SQLStatement longer than 32767 characters,

Reporting Services 2008 Configuration Mistake

To start working with the management side of SQL Server Reporting Services 2008, I decided to set up a report server and report manager. Unfortunately, I made a mistake while setting up my configuration that left me a little perplexed. Here are the steps I took to cause, track down, and solve the issue. Problem: I began by opening the Reporting Services Configuration Manager from the Start Menu. I clicked through each of the menu options and accepted the defaults for any question with a warning symbol, since warning symbol typically designate an action item. After two minutes, all of the warning symbols had disappeared, and I was ready to begin managing my report server. Unfortunately, opening up a browser and trying to open up the report manager resulted in the dreaded " The report server has encountered a configuration error. (rsServerConfigurationError) " message. Sherlock-ing it: I put on my sleuthing hat and went to the log file directory: C:\Program Files\Microsoft