Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 3

Continuing: Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 2

Download SSISProject containing:

  1. DeploymentDemo SSIS Project
  2. SimpleSSISDeploy.ps1

Step 3: Execute SimpleSSISDeploy.ps1

Last and definitely NOT the least, where all the magic happens – SimpleSSISDeploy.ps1.
I put comments where needed, and you can download a zip of all the components, as well as a sample SSIS project at the bottom of this post. 🙂

param ([string]$ConfigurationsPath,
       [string]$IspacPath, 
       [string]$ParametersXml, 
       [string]$Environment)
       
# Store the IntegrationServices Assembly namespace to avoid typing it every time
$ISNamespace = "Microsoft.SqlServer.Management.IntegrationServices"

# Load the IntegrationServices Assembly
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Management.IntegrationServices") | Out-Null;

       
function Main()
{

    # Input validation
    if (-Not(Test-Path($ParametersXml)))
    {
        throw New-Object System.ArgumentException "Parameters.xml not found"
        return
    }
    
    #Get configurations from *.dtproj.user
    [xml]$configFile =  Get-Content $ConfigurationsPath
    $configurationsNode = $configFile.SelectNodes("/DataTransformationsUserConfiguration/Configurations/Configuration")
    
    # Get configuration for $Environment parameter
    $configurationsNode | % {
        if ($_.Name -eq $Environment)
        {
            $serverName = $_.Options.ServerName
            $pathOnServer = $_.Options.PathOnServer
        }
    }   
    
    if ([string]::IsNullOrEmpty($serverName) -or [string]::IsNullOrEmpty($pathOnServer) -or 
        $serverName -eq $null -or $pathOnServer -eq $null )
    {
        throw New-Object System.ArgumentException "Could not connect to Server: $serverName. Does it really exist?"
        return
    }   
    
    # Get catalog, folder and project name from $pathOnServer
    $catalogConnectionString = "Data Source=" + $serverName + ";Initial Catalog=master;Integrated Security=SSPI;"
    $path = $pathOnServer.Split("/", 4)
    $catalogName = $path[1]
    $folderName = $path[2]
    $projectName = $path[3]
    
    if ([string]::IsNullOrEmpty($catalogName) -or 
        [string]::IsNullOrEmpty($folderName) -or 
        [string]::IsNullOrEmpty($projectName))
    {
        throw New-Object System.ArgumentException "Check that $Environment build configuration is set correctly in the SSIS project."
        return
    }
    
    # Connect to the SSIS Server
    $sqlConnection = New-Object System.Data.SqlClient.SqlConnection $catalogConnectionString
    $integrationServices = New-Object $ISNamespace".IntegrationServices" $sqlConnection
    
    if ($integrationServices -eq $null)
    {
        Write-Host "Unable to connect to Integration Services Catalog."
        return
    }
    
    # Get the existing catalog if it exists
    if ($integrationServices.Catalogs.Contains($catalogName)) 
    {
        Write-Host "$catalogName catalog found"
        $catalog = $integrationServices.Catalogs[$catalogName]        
    }
    else
    {
        Write-Host "Could not find "$catalogName" Catalog. Are you sure you have the correct name?"
        return
    }
    
    # Get catalog folder
    if ($catalog.Folders.Contains($folderName))
    {
        Write-Host "$folderName catalog folder found"
        $folder = $catalog.Folders[$folderName]
    }
    else
    {
        Write-Host "Could not find $folderName catalog folder. You are almost there."
        return
    }
    
    # Read the project file, and deploy it to the folder
    [byte[]] $projectFile = [System.IO.File]::ReadAllBytes($IspacPath)
    $project = $folder.DeployProject($projectName, $projectFile)
    
    # Get project
    if ($folder.Projects.Contains($projectName)) 
    {
        Write-Host "$projectName project found"
        $project = $folder.Projects[$projectName]
        
    } else {
        Write-Host "$projectName project not found. Sorry :("
        return 
    }
    
    # Get function parameters from file
    Write-Host "Reading from Parameters.xml"    
    [xml]$file = Get-Content $ParametersXml
    
    Update-Parameters $project $file 
} 

function Update-Parameters($project, $file)
{
    # Update Project Parameters
    $projectParameters = $file.SelectNodes("/SSIS/" + $Environment + "/ProjectParameters/Parameter")

    $projectParameters | % {
        
        $parameter = $_.Name
        if ($parameter -eq $null)
        {
            continue
        }
        
        if ($project.Parameters.Contains($parameter))
        {
            Write-Host "$parameter project parameter found"
            $project.Parameters[$_.Name].Set([Microsoft.SqlServer.Management.IntegrationServices.ParameterInfo+ParameterValueType]::Literal,$_.InnerText)
            Write-Host "$parameter project parameter value updated"
        }
        else
        {
             Write-Host "$parameter project parameter NOT FOUND"
        }
    }
    
    Write-Host "Updating parameters of" $project.Name "project successful"

    # Update Package Parameters
    $packages = $file.SelectNodes("/SSIS/" + $Environment + "/Packages/Package")

    $packages | % {
        
        $packageName = $_.Name
        $parameters = $_.Parameters.ChildNodes
        
        if ($project.Packages.Contains($packageName))
        {
                Write-Host "$packageName package found"
                $ssisPackage = $project.Packages[$packageName]
                
                foreach ($param in $parameters){
                
                    $paramName = $param.Name
                    $paramValue = $param.InnerText
                    
                    if ($ssisPackage.Parameters.Contains($paramName))
                    {
                        Write-Host "$paramName package parameter found"
                        $ssisPackage.Parameters[$paramName].Set([Microsoft.SqlServer.Management.IntegrationServices.ParameterInfo+ParameterValueType]::Literal,$paramValue)
                        Write-Host "$paramName package parameter value updated"
                    }
                    else
                    {
                        Write-Host "$paramName package parameter NOT FOUND"
                    }
                } 
                Write-Host "Updating parameters of $packageName package successful"
                $ssisPackage.Alter() 
                         
        }
        else
        {
            Write-Host "$packageName package NOT FOUND"
        }
        
    }

    $project.Alter()
}

Main

Sample execution code:


.\SimpleSSISDeploy.ps1 -ConfigurationsPath "F:\PROJECTS\SSISProject\DeploymentDemo\DeploymentDemo\DeploymentDemo.dtproj.user" -IspacPath "F:\PROJECTS\SSISProject\DeploymentDemo\DeploymentDemo\bin\Development\DeploymentDemo.ispac" -ParametersXml "F:\PROJECTS\SSISProject\DeploymentDemo\DeploymentDemo\Parameters.xml" -Environment "Development"

And that concludes my solution to an automated SSIS Deployment and configuration management. Feel free to comment if anything is not clear. 🙂

Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 2

Continuing Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 1

Download SSISProject containing:

  1. DeploymentDemo SSIS Project
  2. SimpleSSISDeploy.ps1

Step 2: Add Parameters.xml to the SSIS Project

Before SQL Server 2012, configuration data for SSIS package such as server names, database names, and SMTP servers can be stored in XML files, environment variables, or a separate configuration database. With the new Parameters feature of SSIS catalog, managing configuration data across your environments has never been so intuitive. Instead of  updating a database or a cryptic XML file, someone could simply login to the SSIS catalog via SSMS and configure parameter values manually. Although configuring an SSIS project directly in Production may seem convenient and give anyone a demi-God feeling, it is, however, very risky and is prone to human error.

To continuously integrate our SSIS project, we can automate the task of updating parameter values after the *.ispac is successfully deployed. To do that, we will use an input file, Parameters.xml, to tell our PowerShell script what value to set each parameters with depending on which environment it is deploying to. We will add this file within the SSIS Project itself, so that

  1. It is under source control
  2. Developers will be reminded to prepare it before deployment and
  3. We have all the information we need to execute our SSIS project in one location

Enough of the intro, here’s exactly what you need to do:

  1. Create a Parameters.xml file under the SSIS Project root folder (same folder as *.dtproj file). Paste the code below:
    
    <SSIS>
        <Development>
            <ProjectParameters>
                <Parameter Name="InitialCatalog">AdventureWorksDWH2012</Parameter>
                <Parameter Name="ServerName">SQLSSISDEV</Parameter>
            </ProjectParameters>
            <Packages>
                <Package Name="MainPackage.dtsx">
                    <Parameters>
                        <Parameter Name="SourceFolderRelativePath">\\Source\Folder\RelativePath</Parameter>
                        <Parameter Name="LogFileName">MainPackage.log</Parameter>
                    </Parameters>
                </Package>
            </Packages>
        </Development>
        <Acceptance>
          <ProjectParameters>
            <Parameter Name="InitialCatalog">AdventureWorksDWH2012</Parameter>
            <Parameter Name="ServerName">SQLSSISACC</Parameter>
          </ProjectParameters>
          <Packages>
            <Package Name="MainPackage.dtsx">
              <Parameters>
                <Parameter Name="SourceFolderRelativePath">\\Source\Folder\RelativePath</Parameter>
                <Parameter Name="LogFileName">MainPackage.log</Parameter>
              </Parameters>
            </Package>
          </Packages>
        </Acceptance>
        <Production>
          <ProjectParameters>
            <Parameter Name="InitialCatalog">AdventureWorksDWH2012</Parameter>
            <Parameter Name="ServerName">SQLSSISPRD</Parameter>
          </ProjectParameters>
          <Packages>
            <Package Name="MainPackage.dtsx">
              <Parameters>
                <Parameter Name="SourceFolderRelativePath">\\Source\Folder\RelativePath</Parameter>
                <Parameter Name="LogFileName">MainPackage.log</Parameter>
              </Parameters>
            </Package>
          </Packages>
        </Production>
    </SSIS>
    
    
  2. Add Parameters.xml file as an existing item to the SSIS Project. It will automatically be placed under Miscellaneous folder.

    Parameters

Next: Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 3

Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 1

Let’s start this by describing the scenario with which the approach described below will be useful to you:

  1. You are using SQL Server 2012/2014
  2. You have a separate server/instance for each staging environment (Development, Acceptance, Production, etc.)
  3. You would like to automate SSIS project deployments to your target environments
  4. You would like to automate managing SSIS package configurations (e.g. server names, database names, etc.) on all environments without using a separate configuration database

If you nod your way through the 4 points above, then take the time to read my solution below. It’s long, but I promise you will find some points mentioned helpful if not the entire thing.

Pre-requisites

I assume that you are familiar with the new Project Deployment Model feature of SSIS 2012/2014. You should also understand how Parameters are utilized with an SSIS Project. You should not be afraid of PowerShell and XML, for neither of them bite and both are very easy to learn. Note that I am not going to use the new Environment Variables feature because it assumes that all environments are deployed to a single server  – which is not the usual case for enterprises or any medium to large scale company for that matter.

The Solution in Summary

My solution has 3 components:

  1. PowerShell script that deploys an *.ispac file to an SSIS Catalog and updates Project and Package Parameters
  2. SSIS Project build configuration to store the SSIS Catalog information
  3. Additional XML file within the SSIS project to store parameter values

Download SSISProject containing:

  1. DeploymentDemo SSIS Project
  2. SimpleSSISDeploy.ps1

Step 1: Setup the SSIS Project’s Build Configuration

We will store the target SSIS Catalog information in the SSIS Project itself so that our script can find on its own where to deploy the *.ispac file. To do that, open the SSIS project then..

  1. In Solution Explorer, right-click on the Visual Studio solution and click Properties.
  2. Solution Property Page dialog opens.
  3. Under Configuration Properties, select Configuration.
  4. Click Configuration Manager at the upper right corner of the window. Configuration Manager dialog opens.
  5. Click the dropdown list under Active solution configuration.
  6. Select New. New Solution Configuration dialog opens.
  7. In the Name field, enter one of your environments. Check “Create new project configurations”. Click OK.
  8. Repeat steps 4 – 6 until you have created solutions and project configurations for all  your environments.
  9. Click Close. Click OK.

Now, we are ready to fill our build configuration for each environment.

  1. In Solution Explorer, right-click on the SSIS project and click Properties. 
  2. Propert Page dialog opens.
  3. Under Configuration Properties, click Deployment. Configuration dropdown is now enabled.
  4. Select one environment created earlier.
  5. In the main pane under Deployment Model (Project), fill the Server Name and Server Project Path details. Click Apply.
  6. Repeat steps 4 – 5 for all environments. Click OK to close the Property Pages window.

Finally, open the folder containing the SSIS project and locate .dtproj.user file. If you can’t see it, modify Folder Options to see the hidden files. The file should look like this:


<?xml version="1.0" encoding="utf-8"?>
<DataTransformationsUserConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <Configurations>
   <Configuration>
     <Name>Development</Name>
<options>
       <ServerName>SQLSSISDEV</ServerName>
       <PathOnServer>/SSISDB/LAB/DeploymentDemo</PathOnServer>
       <UserIDs />
       <UserPasswords />
       <OfflineMode>false</OfflineMode>
       <ProgressReporting>true</ProgressReporting>
       <ParameterConfigurationSensitiveValues />
     </Options>
   </Configuration>
   <Configuration>
     <Name>Acceptance</Name>
<options>
        <ServerName>SQLSSISACC</ServerName>
        <PathOnServer>/SSISDB/LAB/DeploymentDemo</PathOnServer>
        <UserIDs />
        <UserPasswords />
        <OfflineMode>false</OfflineMode>
        <ProgressReporting>true</ProgressReporting>
        <ParameterConfigurationSensitiveValues />
      </Options>
  </Configuration>
  <Configuration>
     <Name>Production</Name>
<options>
        <ServerName>SQLSSISPRD</ServerName>
        <PathOnServer>/SSISDB/LAB/DeploymentDemo</PathOnServer>
        <UserIDs />
        <UserPasswords />
        <OfflineMode>false</OfflineMode>
        <ProgressReporting>true</ProgressReporting>
        <ParameterConfigurationSensitiveValues />
      </Options>
  </Configuration>
 </Configurations>
</DataTransformationsUserConfiguration>

Next: Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 2

How to “downgrade” SQL Server Integration Services 2014 packages to 2012

The Problem

A few months back, I built an SSIS 2014 project using Visual Studio 2013 that is supposed to serve as a template for future data warehouse projects. Our servers are still in SQL Server 2012, but the upgrade has been approved and so it made sense to create the template project in VS 2013. Until one day, the template was needed for a new project, with the promised SQL Server 2014 upgrade not yet in sight. Since SSDT-BI 2013 is not backward-compatible with SQL Server 2012, I then had to downgrade the SSIS 2014 project so that people can use it using Visual Studio 2012 and deploy it to SQL Server 2012. There are no tools from Microsoft, no StackOverflow forum that gave the perfect answer and mattmasson has kept quiet about the topic. 😦

The Workaround (Summary)

Based on my observation (only  Microsoft can verify), excluding the upgraded components, there are three changes between the XML files of SSDT-BI 2012 and SSDT-BI 2014 dtsx packages.

  1. PackageFormatVersion

SSDT-BI 2012 VS 2012: DTS:Name=”PackageFormatVersion”>6

SSDT-BI 2014 VS 2013: DTS:Name=”PackageFormatVersion”>8

  1. The values used in ExecutableType properties

SSDT-BI 2012 VS 2012: DTS:ExecutableType=”SSIS.Package.3″

SSDT-BI 2014 VS 2013: DTS:ExecutableType=”Microsoft.Package”

  1. The values used in componentClassID attributes

SSDT-BI 2012 VS 2012: componentClassID=”{49928E82-9C4E-49F0-AABE-3812B82707EC}”

SSDT-BI 2014 VS 2013: componentClassID=”Microsoft.DerivedColumn”

Have you guessed the fix already? You got it! It’s only a matter of find (all) and replace! Open the dtsx file in your favorite text editor (mine is  Notepad++) and replace the values used in the DTSX2 Version 2014/01 schema with the ones in DTSX  Version 2012/01 schema. 🙂 Now, there is some “manual labor” required, because you have to manually search for the componentClassIDs and ExecutableTypes in the DTSX  Version 2012/01 schema. Here are the ExecutableTypes and componentClassIDs I encountered so far. Click Replace.xls for an Excel version.

replace

 The Workaround (Step by Step)

  1. If you haven’t done so, open DTSX Version 2012/01 schema
  2. Open the SSDT-BI VS 2013 DTSX package in a text editor
  3. Search for PackageFormatVersion. Change the value from 8 to 6
  4. Replace all instances of Microsoft.Package with SSIS.Package.3
  5. Replace all instances of Microsoft.Pipeline with SSIS.Pipeline.3
  6. Search for all instances of DTS:ExecutableType=”
    unnamed
  7. For each ExecutableType value (Microsoft.ExecuteSQLTask, etc.), go to the DTSX Version 2012/01 schema and search under
    <xs:simpleType name="AnyNonPackageExecutableTypeExecutableTypeEnum">

    for the correct ExecutableType. For example, Microsoft.ExecuteSQLTask in SSIS 2014 should be Microsoft.SqlServer.Dts.Tasks.ExecuteSQLTask.ExecuteSQLTask, Microsoft.SqlServer.SQLTask, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc9 in SSIS 2012.

  8. Replace ALL instances (including CreationName attributes) of the ExecutableType value with the one you got from Step 7
  9. Search for all instances of componentClassID=”Microsoft
    componentClassID
  10. For each componentClassID value (Microsoft.DerivedColumn, Microsoft.Lookup, etc.), go to the DTSX Version 2012/01 schema and search under
    <xs:simpleType name="PipelineComponentComponentClassIDEnum">

    for the corresponding GUID.

  11. Replace ALL instances (including CreationName attributes) of the componentClassID value with the one from Step 10.
  12. Now open the package in Visual Studio 2012. If your package does not have upgraded components, then it should open by now. Otherwise, read further.

The Exception

Some components, such as the Script Component in the Data Flow Task, have been upgraded from SSDT-BI 2012 to SSDT-BI 2014. In this case, it’s not only the property / attribute value that is different, but also the structure of the XML node. In this case, replace the Version 2014/01 schema value (e.g. componentClassID=”Microsoft.ManagedComponentHost”)  with any valid DTSX Version 2012/01 componentClassID. Then open the package in VS 2012. The component will be marked as with error, that’s OK. If you have followed all the steps above, your package should at least load the designer by now. Now remove this erroneous component and re-do it in VS 2012 itself.

Conclusion

I  was able downgrade SSIS 2014 to SSIS 2012 packages by modifying the XML file manually. I downgraded 15 packages in 4 hours including finding the workaround described above. It should be faster if you already know what to do. This approach seems more reasonable than re-creating all the packages I designed and developed for weeks with all their DFTs and expressions. No, thank you. Let me know if you encounter any other exceptions or if any of the steps above is not clear.

[SSIS] Lookup Transformation – Full Cache option

Level: Intermediate

Lookup transformation is an essential ingredient of a Data Flow task accessing multiple database servers. I always use the Full Cache option when the reference table is small enough to be stored in memory. However, this caching option needs more than setting up your Lookup task to work. Here are few things to check when your Lookup transformation fails to detect a match you are expecting it to detect:

Case sensitivity – the columns being compared must have exactly the same case. Otherwise, the match will fail.

Solution: Use UPPER() OR LOWER() on BOTH the Lookup AND the pipeline columns

Continue reading “[SSIS] Lookup Transformation – Full Cache option”