Friday, February 7, 2014

cannot convert between unicode and non-unicode string data types

[OLE DB Source [628]] Error: Column "COLUMN_NAME" cannot convert between unicode and non-unicode string data types


When creating a SSIS package, this kind of error pops up often.

This is caused by different data types used in different applications.


For instance,
  SSIS SQL Oracle
Unicode DT_WSTR nchar, nvarchar nvarchar2
Non-Unicode DT_STR char, varchar varchar2
There are couple of ways to convert data types

1) data conversion

     Very easy to convert. Data types can be changed in Data Conversion Transformation Editor.

2) Oracle connection sometimes requires data conversion within that task. Which means, it errors out even before data conversion

     In this case, cast columns with unistr in a sql command
     
     Select unistr(location_code) as location_code, unistr(last_posted_date) as last_posted_date
     from T_LOCATION

3) Lastly, datatype can be changed within OLE DB Source/destination 

     go to OLE DB source or OLE DB destination advance editor and change data types in "Input and                Output Properties tab"



Translation Maps

References

Tuesday, February 4, 2014

How to connect Oracle from SSIS and deploy it as a sql job

Things to remember before creating a SSIS SQL job.

* 32 bit and 64 bit Oracle provider need to be installed on the SQL server.
* Visual Studio is 32 bit - make sure check "User 32 bit runtime" box in Execution options in Job step properties.
* It is better to use Oracle Provider for OLEDB than Microsoft Provider for Oracle. Although many people say do not use Microsoft Provider for Oracle since it's an old driver and doesn't support 64bit, I was able to use both with no problem. However, it all depends on your environment. So, it is highly recommended to go with Oracle Provider for OLEDB.

1) For Oracle connection

a) Using Microsoft OLE DB Provider for Oracle

Use server name and put the user name and password for oracle


Server name is like this.  *This is the same address as TNS if you created one
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(Host="Host Address")(Port="Port number")))(CONNECT_DATA=(SERVICE_NAME="service name")))

b) Using Oracle Provider for OLE DB


Same as above and make sure that check save password box.


Because of Oracle log-on information need to be included with the package itself we need to pick the right package protection level in order to retrieve log-on information.

2) Package protection level



a) DontSaveSensitive
  • Create a SSISConfigurationsStorage table



  • note (create a view table if multiple accounts have to be used for dev, QA, UAT, etc. Since SSISConfigurationsStorage table has a ServiceAccount column, a view table will show the packagePath accordingly only to those service accounts)
  • create two package configurations - environment variable and SQL server config.
  • create a same environment variable on the server
  • Use FilePathDestination Deployment type to deploy SSIS package. (It's nice if it can be deployed to the server directly. If it's not the case than deploy to your local machine and connect to integration service and import the SSIS package file from there)
  • Create a job in SQL

  • create a step and check User 32bit runtime box in execution options tab

  • And it's done.  The good thing about this method is that oracle credentials are stored in configuration table with connection string. It's easy to control multiple packages and service accounts in one place. Of course there are other ways of doing this but for simplicity and less maintenance purpose I think this is the best method. For instance, if the package was built with EncryptSensitiveWithPassword option, each time when someone wants to access to the package you have to type the password and need to add decrypt option in command line tab. 
References


Helpful links

Tuesday, February 12, 2013

how to use page breaks to keep a table in one page

This is how to keep a table in one page by counting row number with custom code.

Here I have three row groups - region, district, and store


For a region group, I can simply add 'Between each instance of group' page break.

However, district group has a problem when it has to break down into two pages because of number of stores in that group.

============================================================
Best thing I could do about it was to add a custom code.

dim total as integer = 0
dim regionID as integer = 0

function getRowCount(i as Integer, r as Integer ) as boolean
if regionID <> r then
regionID = r
total = 0
end if
total = total + i
if total <= 45 then
return true
else 
total = i
return false
end if
end function
===========================================================

and use group property variables with expression


and then use properties to set page break.


and that's it. This will prevent a table splitting in two pages.

Reporting Server Report

How to pull reporting server reports?

I was once asked to collect all reports information from reporting servers.

And this is how I came up with.

========================================================

select c.Name as ReportName, d.name as DataSourceName, convert(nvarchar(10), max(e.timestart), 101) as LastAccessed, c.Path, COUNT(e.timestart) as TotalNum, c.Hidden
--,
--(case
--when c.Type = 1 then 'Folder'
--when c.type = 2 then 'Report'
--when c.Type = 3 then 'Resources'
--when c.Type = 4 then 'Linked Report'
--when c.Type = 5 then 'Data Source'
--when c.Type = 6 then 'Report Model'
--when c.Type = 7 then 'Report Part'
--when c.Type = 8 then 'Shared Dataset'
--end) as ReportType
from Catalog c
left join
DataSource d
on c.ItemID = d.ItemID
left join
ExecutionLog e
on c.ItemID = e.ReportID
where c.type = 2
group by c.Name, d.Name, c.Path, c.Hidden--, c.Type
order by c.path, c.Name
========================================================

*Problem with this query is that it's not actually pulling DataSourceName correctly sometimes.

If someone creates a report and that report doesn't have the same DataSourceName as on the server, the query above will pull 'DataSource1' as a DataSourceName.

So I had to pull all DataSourceName from reporting server and use the next query and put them into Excel to get the right DataSourceName.

This query will parse XML and give Connection string.
========================================================
;WITH XMLNAMESPACES  -- XML namespace def must be the first in with clause. 
    (DEFAULT 'http://schemas.microsoft.com/sqlserver/reporting/2006/03/reportdatasource' 
            ,'http://schemas.microsoft.com/SQLServer/reporting/reportdesigner' 
     AS rd) 
,SDS AS 
    (SELECT SDS.name AS SharedDsName 
           ,SDS.[Path] 
           ,CONVERT(xml, CONVERT(varbinary(max), content)) AS DEF 
     FROM dbo.[Catalog] AS SDS 
     WHERE SDS.Type = 5)     -- 5 = Shared Datasource 

SELECT CON.[Path] 
      ,CON.SharedDsName 
      ,CON.ConnString 
FROM 
    (SELECT SDS.[Path] 
           ,SDS.SharedDsName 
           ,DSN.value('ConnectString[1]', 'varchar(150)') AS ConnString 
     FROM SDS 
          CROSS APPLY  
          SDS.DEF.nodes('/DataSourceDefinition') AS R(DSN)
     ) AS CON 
-- Optional filter: 
-- WHERE CON.ConnString LIKE '%Initial Catalog%=%TFS%' 
ORDER BY CON.[Path] 
        ,CON.SharedDsName;

I googled and found this from Olaf Helper
<http://gallery.technet.microsoft.com/scriptcenter/List-connection-strings-of-1a9a9adc>

========================================================

According to Microsoft, they 'do not document or support querying any Report Catalog tables'

You can check this forum.
<http://social.msdn.microsoft.com/Forums/en-US/sqlreportingservices/thread/60dd3392-42d8-4dc4-b8e6-15e9aeaad29e/>

========================================================
or here's a simpler version


select
  c.name ReportName,
  d.name DataSource,
  c2.Name SharedDataSourceName,
  c.Path ReportPath,
  c2.Path DataSourcePath,
  convert(nvarchar(10), max(e.timestart), 101) as LastAccessed,
  COUNT(e.timestart) as TotalNum, c.Hidden
from
Catalog c
left join DataSource d
on c.ItemID = d.ItemID
left join Catalog c2
on d.Link = c2.ItemID
left join
ExecutionLog e
on c.ItemID = e.ReportID
where c.Type = 2 and c.Path not like '%BackUp%' and c.Name not like '%old%' --and c.Name like '%Sub%'-- and e.TimeStart > '6/1/2012'
group by c.Name, d.name, c2.Name, c.Path, c.Hidden, c2.Path
order by c2.Path, c.Path


or this


SELECT
    DS.Name AS DatasourceName,
    C.Name AS DependentItemName,
    C.Path AS DependentItemPath,
    COUNT(EL.timestart) as AccessedSinceJune2012
FROM
    ReportServer.dbo.Catalog AS C
        INNER JOIN
    ReportServer.dbo.Users AS CU
        ON C.CreatedByID = CU.UserID
        INNER JOIN
    ReportServer.dbo.Users AS MU
        ON C.ModifiedByID = MU.UserID
        LEFT OUTER JOIN
    ReportServer.dbo.SecData AS SD
        ON C.PolicyID = SD.PolicyID AND SD.AuthType = 1
        INNER JOIN
    ReportServer.dbo.DataSource AS DS
        ON C.ItemID = DS.ItemID
        LEFT JOIN
    ReportServer.dbo.ExecutionLog AS EL
ON C.ItemID = EL.ReportID
       
WHERE
    DS.Name IS NOT NULL and c.Name not like '%old%' and c.Name not like '%test' and c.path not like '%BackUp%' and EL.TimeStart > '6/1/2012'
GROUP BY
DS.Name, C.Name, C.Path
ORDER BY
    DS.Name;


*I just added executionlog in order to get how many times it was accessed

Original post is from here
< http://stackoverflow.com/questions/9638431/listing-all-data-sources-and-their-dependencies-reports-items-etc-in-sql-ser>


Monday, August 13, 2012

found some useful sql tips


·      Insert multiple rows
insert into tablename (col1, col2)
select ‘this1’, ‘that1’
union all
select ‘this2’, ‘that2’
union all
select ‘this3’, ‘that3’
union all
select ‘this4’, ‘that4’
From Personal Notes of Pinal Dave


·         Conditional Where Clause
select *
from   dbo.tblConditionalWhereClause
where  ((Businessdate = @BusinessDate) and (_NODEID_ = @nodeID))
              or     ((@BusinessDate is null) and (_NODEID_ = @nodeID))
              or     ((@nodeID is null) and (Businessdate = @BusinessDate))
              or  ((@BusinessDate is null) and (@nodeID is null))
logical where clause (Not A) or B means if A then B

create a child package and pass variables with this Execute Package Task

Now we need to create a child package.

1) create a new package and create new package variables

2) right click on the design screen and click Package Configurations

3) On the pop up window, enable package configurations and click Add...

4) click next and change Configuration type to Parent package variable and type a Parent variable

5) on the next screen, expand Variables and variable name(this time "logFileName") and expand its properties and click Value and click Next

6) then click next and you will see a new configuration on the Package Configurations Organizer window

7) repeat this if you have more than one variable to pass from parent to child package.
(In my project, I had to pass 3 variables - LogFileName, TxtFileName, and LogSource)

8) then build this package.

9) go back to parent package and modify Execute Package Task 

10) Location can be SQL server or file system depending on where you built your child package. 


Wednesday, August 8, 2012

SSIS 2008 - flat file source look up other flat file with foreach loop

I don't know what this title should be but here's simplified objectives


1. Pull from a specified directory all files matching a format of “007C0*.log
2. From each error log, you must retrieve the line number from the contents of the error log and the error message
3. Using the file name, pull from  another directory the file that has the same base name, but with a “txt” extension
4. Using the Line Number from the error log, pull the line data from the text file.
5. Use both the information from the error log and the upload file to populate the table

log file format (007C0*.log)

text file format (logfilename_datetimestamp.txt)

Objective 1 and 2 can be done in one foreach loop.
For objective 3, I needed one more foreach loop since text file consists of full log file name and datetimestamp.
For objective 4, I had to put a data into somewhere but I was asked not to put in our database so I had to use cache transform. That means, I had to create two packages - parent and child since cache transform cannot be placed in a for loop(connection problem).

So lets start!

1) create five variables.

2) put a foreach loop container and double click to open up an editor.
click Collection and click + sign to expand Expressions and click ... to put new values
3) Pick Directory and FileSpec  from Property column drop down menu.
and Click ... to add expression.
(In this example, FileSpec is a filename and  Directory is the logfilesource.)

4) click Variable Mappings tab and pick [User::LogFileName] and close an editor.

5) add a Script Task inside Foreach Loop Container. (need to get a log file name without a path and file extension)
Click ReadWriteVariables row and add [User::LogFileName] and click Edit Script

5.1) simple code
         public void Main()
        {   
            String str = Dts.Variables["User::LogFileName"].Value.ToString();            
            String logfilename = str.Substring(3, str.IndexOf(".")-3);
            Dts.Variables["User::LogFileName"].Value = logfilename;            
            Dts.TaskResult = (int)ScriptResults.Success;
        }

* the reason why I used "3"s in str.substring is that the path is "C:\". If your path(directory) is like "C:\Users\asung\Documents\Visual Studio 2008\Projects\Integration Services Project3" then you need to use a different number. 


6) add a new Foreach Loop Container inside and set it up like the one before
But this time FileSpec Expression should be @[User::LogFileName] + "*.txt"

7) click Variable Mappings tab and pick [User::TxtFileName] and close an editor.

8) add a Script Task inside an inner Foreach Loop Container and repeat step 5 with [User::TxtFileName]

8.1) just like 5.1
        public void Main()
        {
            String str = Dts.Variables["User::TxtFileName"].Value.ToString();
            Dts.Variables["User::TxtFileName"].Value = str.Substring(11);
            Dts.TaskResult = (int)ScriptResults.Success;
        }


9) add an Execute Package Task


Next is to create a child package and pass variables with this Execute Package Task.