How to Implement Slowly Changing Dimension Type 2 in SQL Server Analysis Services

The other day somebody asked me how I could implement Slowly Changing Dimension Type 2 (SCD Type 2) in SSAS. I wasn’t able to give the answer right there and then, so I decided to create a demo video to answer his question and as a note to myself for future uses.

This is my first ever demo video by the way and I agree that it could have been better in many parts. I do hope that despite my amateur recording and editing skills, I am still able to send the message across. 🙂

Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 3

Continuing: Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 2

Download SSISProject containing:

  1. DeploymentDemo SSIS Project
  2. SimpleSSISDeploy.ps1

Step 3: Execute SimpleSSISDeploy.ps1

Last and definitely NOT the least, where all the magic happens – SimpleSSISDeploy.ps1.
I put comments where needed, and you can download a zip of all the components, as well as a sample SSIS project at the bottom of this post. 🙂

param ([string]$ConfigurationsPath,
       [string]$IspacPath, 
       [string]$ParametersXml, 
       [string]$Environment)
       
# Store the IntegrationServices Assembly namespace to avoid typing it every time
$ISNamespace = "Microsoft.SqlServer.Management.IntegrationServices"

# Load the IntegrationServices Assembly
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Management.IntegrationServices") | Out-Null;

       
function Main()
{

    # Input validation
    if (-Not(Test-Path($ParametersXml)))
    {
        throw New-Object System.ArgumentException "Parameters.xml not found"
        return
    }
    
    #Get configurations from *.dtproj.user
    [xml]$configFile =  Get-Content $ConfigurationsPath
    $configurationsNode = $configFile.SelectNodes("/DataTransformationsUserConfiguration/Configurations/Configuration")
    
    # Get configuration for $Environment parameter
    $configurationsNode | % {
        if ($_.Name -eq $Environment)
        {
            $serverName = $_.Options.ServerName
            $pathOnServer = $_.Options.PathOnServer
        }
    }   
    
    if ([string]::IsNullOrEmpty($serverName) -or [string]::IsNullOrEmpty($pathOnServer) -or 
        $serverName -eq $null -or $pathOnServer -eq $null )
    {
        throw New-Object System.ArgumentException "Could not connect to Server: $serverName. Does it really exist?"
        return
    }   
    
    # Get catalog, folder and project name from $pathOnServer
    $catalogConnectionString = "Data Source=" + $serverName + ";Initial Catalog=master;Integrated Security=SSPI;"
    $path = $pathOnServer.Split("/", 4)
    $catalogName = $path[1]
    $folderName = $path[2]
    $projectName = $path[3]
    
    if ([string]::IsNullOrEmpty($catalogName) -or 
        [string]::IsNullOrEmpty($folderName) -or 
        [string]::IsNullOrEmpty($projectName))
    {
        throw New-Object System.ArgumentException "Check that $Environment build configuration is set correctly in the SSIS project."
        return
    }
    
    # Connect to the SSIS Server
    $sqlConnection = New-Object System.Data.SqlClient.SqlConnection $catalogConnectionString
    $integrationServices = New-Object $ISNamespace".IntegrationServices" $sqlConnection
    
    if ($integrationServices -eq $null)
    {
        Write-Host "Unable to connect to Integration Services Catalog."
        return
    }
    
    # Get the existing catalog if it exists
    if ($integrationServices.Catalogs.Contains($catalogName)) 
    {
        Write-Host "$catalogName catalog found"
        $catalog = $integrationServices.Catalogs[$catalogName]        
    }
    else
    {
        Write-Host "Could not find "$catalogName" Catalog. Are you sure you have the correct name?"
        return
    }
    
    # Get catalog folder
    if ($catalog.Folders.Contains($folderName))
    {
        Write-Host "$folderName catalog folder found"
        $folder = $catalog.Folders[$folderName]
    }
    else
    {
        Write-Host "Could not find $folderName catalog folder. You are almost there."
        return
    }
    
    # Read the project file, and deploy it to the folder
    [byte[]] $projectFile = [System.IO.File]::ReadAllBytes($IspacPath)
    $project = $folder.DeployProject($projectName, $projectFile)
    
    # Get project
    if ($folder.Projects.Contains($projectName)) 
    {
        Write-Host "$projectName project found"
        $project = $folder.Projects[$projectName]
        
    } else {
        Write-Host "$projectName project not found. Sorry :("
        return 
    }
    
    # Get function parameters from file
    Write-Host "Reading from Parameters.xml"    
    [xml]$file = Get-Content $ParametersXml
    
    Update-Parameters $project $file 
} 

function Update-Parameters($project, $file)
{
    # Update Project Parameters
    $projectParameters = $file.SelectNodes("/SSIS/" + $Environment + "/ProjectParameters/Parameter")

    $projectParameters | % {
        
        $parameter = $_.Name
        if ($parameter -eq $null)
        {
            continue
        }
        
        if ($project.Parameters.Contains($parameter))
        {
            Write-Host "$parameter project parameter found"
            $project.Parameters[$_.Name].Set([Microsoft.SqlServer.Management.IntegrationServices.ParameterInfo+ParameterValueType]::Literal,$_.InnerText)
            Write-Host "$parameter project parameter value updated"
        }
        else
        {
             Write-Host "$parameter project parameter NOT FOUND"
        }
    }
    
    Write-Host "Updating parameters of" $project.Name "project successful"

    # Update Package Parameters
    $packages = $file.SelectNodes("/SSIS/" + $Environment + "/Packages/Package")

    $packages | % {
        
        $packageName = $_.Name
        $parameters = $_.Parameters.ChildNodes
        
        if ($project.Packages.Contains($packageName))
        {
                Write-Host "$packageName package found"
                $ssisPackage = $project.Packages[$packageName]
                
                foreach ($param in $parameters){
                
                    $paramName = $param.Name
                    $paramValue = $param.InnerText
                    
                    if ($ssisPackage.Parameters.Contains($paramName))
                    {
                        Write-Host "$paramName package parameter found"
                        $ssisPackage.Parameters[$paramName].Set([Microsoft.SqlServer.Management.IntegrationServices.ParameterInfo+ParameterValueType]::Literal,$paramValue)
                        Write-Host "$paramName package parameter value updated"
                    }
                    else
                    {
                        Write-Host "$paramName package parameter NOT FOUND"
                    }
                } 
                Write-Host "Updating parameters of $packageName package successful"
                $ssisPackage.Alter() 
                         
        }
        else
        {
            Write-Host "$packageName package NOT FOUND"
        }
        
    }

    $project.Alter()
}

Main

Sample execution code:


.\SimpleSSISDeploy.ps1 -ConfigurationsPath "F:\PROJECTS\SSISProject\DeploymentDemo\DeploymentDemo\DeploymentDemo.dtproj.user" -IspacPath "F:\PROJECTS\SSISProject\DeploymentDemo\DeploymentDemo\bin\Development\DeploymentDemo.ispac" -ParametersXml "F:\PROJECTS\SSISProject\DeploymentDemo\DeploymentDemo\Parameters.xml" -Environment "Development"

And that concludes my solution to an automated SSIS Deployment and configuration management. Feel free to comment if anything is not clear. 🙂

Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 2

Continuing Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 1

Download SSISProject containing:

  1. DeploymentDemo SSIS Project
  2. SimpleSSISDeploy.ps1

Step 2: Add Parameters.xml to the SSIS Project

Before SQL Server 2012, configuration data for SSIS package such as server names, database names, and SMTP servers can be stored in XML files, environment variables, or a separate configuration database. With the new Parameters feature of SSIS catalog, managing configuration data across your environments has never been so intuitive. Instead of  updating a database or a cryptic XML file, someone could simply login to the SSIS catalog via SSMS and configure parameter values manually. Although configuring an SSIS project directly in Production may seem convenient and give anyone a demi-God feeling, it is, however, very risky and is prone to human error.

To continuously integrate our SSIS project, we can automate the task of updating parameter values after the *.ispac is successfully deployed. To do that, we will use an input file, Parameters.xml, to tell our PowerShell script what value to set each parameters with depending on which environment it is deploying to. We will add this file within the SSIS Project itself, so that

  1. It is under source control
  2. Developers will be reminded to prepare it before deployment and
  3. We have all the information we need to execute our SSIS project in one location

Enough of the intro, here’s exactly what you need to do:

  1. Create a Parameters.xml file under the SSIS Project root folder (same folder as *.dtproj file). Paste the code below:
    
    <SSIS>
        <Development>
            <ProjectParameters>
                <Parameter Name="InitialCatalog">AdventureWorksDWH2012</Parameter>
                <Parameter Name="ServerName">SQLSSISDEV</Parameter>
            </ProjectParameters>
            <Packages>
                <Package Name="MainPackage.dtsx">
                    <Parameters>
                        <Parameter Name="SourceFolderRelativePath">\\Source\Folder\RelativePath</Parameter>
                        <Parameter Name="LogFileName">MainPackage.log</Parameter>
                    </Parameters>
                </Package>
            </Packages>
        </Development>
        <Acceptance>
          <ProjectParameters>
            <Parameter Name="InitialCatalog">AdventureWorksDWH2012</Parameter>
            <Parameter Name="ServerName">SQLSSISACC</Parameter>
          </ProjectParameters>
          <Packages>
            <Package Name="MainPackage.dtsx">
              <Parameters>
                <Parameter Name="SourceFolderRelativePath">\\Source\Folder\RelativePath</Parameter>
                <Parameter Name="LogFileName">MainPackage.log</Parameter>
              </Parameters>
            </Package>
          </Packages>
        </Acceptance>
        <Production>
          <ProjectParameters>
            <Parameter Name="InitialCatalog">AdventureWorksDWH2012</Parameter>
            <Parameter Name="ServerName">SQLSSISPRD</Parameter>
          </ProjectParameters>
          <Packages>
            <Package Name="MainPackage.dtsx">
              <Parameters>
                <Parameter Name="SourceFolderRelativePath">\\Source\Folder\RelativePath</Parameter>
                <Parameter Name="LogFileName">MainPackage.log</Parameter>
              </Parameters>
            </Package>
          </Packages>
        </Production>
    </SSIS>
    
    
  2. Add Parameters.xml file as an existing item to the SSIS Project. It will automatically be placed under Miscellaneous folder.

    Parameters

Next: Automate SSIS 2012 Project Deployment and Configuration Management using PowerShell – Part 3

How to “downgrade” SQL Server Integration Services 2014 packages to 2012

The Problem

A few months back, I built an SSIS 2014 project using Visual Studio 2013 that is supposed to serve as a template for future data warehouse projects. Our servers are still in SQL Server 2012, but the upgrade has been approved and so it made sense to create the template project in VS 2013. Until one day, the template was needed for a new project, with the promised SQL Server 2014 upgrade not yet in sight. Since SSDT-BI 2013 is not backward-compatible with SQL Server 2012, I then had to downgrade the SSIS 2014 project so that people can use it using Visual Studio 2012 and deploy it to SQL Server 2012. There are no tools from Microsoft, no StackOverflow forum that gave the perfect answer and mattmasson has kept quiet about the topic. 😦

The Workaround (Summary)

Based on my observation (only  Microsoft can verify), excluding the upgraded components, there are three changes between the XML files of SSDT-BI 2012 and SSDT-BI 2014 dtsx packages.

  1. PackageFormatVersion

SSDT-BI 2012 VS 2012: DTS:Name=”PackageFormatVersion”>6

SSDT-BI 2014 VS 2013: DTS:Name=”PackageFormatVersion”>8

  1. The values used in ExecutableType properties

SSDT-BI 2012 VS 2012: DTS:ExecutableType=”SSIS.Package.3″

SSDT-BI 2014 VS 2013: DTS:ExecutableType=”Microsoft.Package”

  1. The values used in componentClassID attributes

SSDT-BI 2012 VS 2012: componentClassID=”{49928E82-9C4E-49F0-AABE-3812B82707EC}”

SSDT-BI 2014 VS 2013: componentClassID=”Microsoft.DerivedColumn”

Have you guessed the fix already? You got it! It’s only a matter of find (all) and replace! Open the dtsx file in your favorite text editor (mine is  Notepad++) and replace the values used in the DTSX2 Version 2014/01 schema with the ones in DTSX  Version 2012/01 schema. 🙂 Now, there is some “manual labor” required, because you have to manually search for the componentClassIDs and ExecutableTypes in the DTSX  Version 2012/01 schema. Here are the ExecutableTypes and componentClassIDs I encountered so far. Click Replace.xls for an Excel version.

replace

 The Workaround (Step by Step)

  1. If you haven’t done so, open DTSX Version 2012/01 schema
  2. Open the SSDT-BI VS 2013 DTSX package in a text editor
  3. Search for PackageFormatVersion. Change the value from 8 to 6
  4. Replace all instances of Microsoft.Package with SSIS.Package.3
  5. Replace all instances of Microsoft.Pipeline with SSIS.Pipeline.3
  6. Search for all instances of DTS:ExecutableType=”
    unnamed
  7. For each ExecutableType value (Microsoft.ExecuteSQLTask, etc.), go to the DTSX Version 2012/01 schema and search under
    <xs:simpleType name="AnyNonPackageExecutableTypeExecutableTypeEnum">

    for the correct ExecutableType. For example, Microsoft.ExecuteSQLTask in SSIS 2014 should be Microsoft.SqlServer.Dts.Tasks.ExecuteSQLTask.ExecuteSQLTask, Microsoft.SqlServer.SQLTask, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc9 in SSIS 2012.

  8. Replace ALL instances (including CreationName attributes) of the ExecutableType value with the one you got from Step 7
  9. Search for all instances of componentClassID=”Microsoft
    componentClassID
  10. For each componentClassID value (Microsoft.DerivedColumn, Microsoft.Lookup, etc.), go to the DTSX Version 2012/01 schema and search under
    <xs:simpleType name="PipelineComponentComponentClassIDEnum">

    for the corresponding GUID.

  11. Replace ALL instances (including CreationName attributes) of the componentClassID value with the one from Step 10.
  12. Now open the package in Visual Studio 2012. If your package does not have upgraded components, then it should open by now. Otherwise, read further.

The Exception

Some components, such as the Script Component in the Data Flow Task, have been upgraded from SSDT-BI 2012 to SSDT-BI 2014. In this case, it’s not only the property / attribute value that is different, but also the structure of the XML node. In this case, replace the Version 2014/01 schema value (e.g. componentClassID=”Microsoft.ManagedComponentHost”)  with any valid DTSX Version 2012/01 componentClassID. Then open the package in VS 2012. The component will be marked as with error, that’s OK. If you have followed all the steps above, your package should at least load the designer by now. Now remove this erroneous component and re-do it in VS 2012 itself.

Conclusion

I  was able downgrade SSIS 2014 to SSIS 2012 packages by modifying the XML file manually. I downgraded 15 packages in 4 hours including finding the workaround described above. It should be faster if you already know what to do. This approach seems more reasonable than re-creating all the packages I designed and developed for weeks with all their DFTs and expressions. No, thank you. Let me know if you encounter any other exceptions or if any of the steps above is not clear.