header
header Register : : Login header
header
divider
menuleft
menuright
submenu
left

[August 25th, 2008] Check the home page regarding PowerShell related news from a brand new sponsor: Idera

Subject: Importing Emails into SQL Server DB
Prev Next
You are not authorized to post a reply.

Author Messages
gsxrian99User is Offline
New Member
New Member
Posts:7

07/12/2008 7:01 AM  

I am looking to automate an import process using powershell.

My goal is to automatically have emails that come into my Outlook 2007 Inbox (XP) get imported into my SQL Server 2005 database. (Windows server 2003)

I also have windows server 2008 and sql server 2008 as an option, but am working thru some install issues with sql 2008 at the moment:(

Is Powershell the best way to do this?

Thanks

  

 

halr9000User is Offline
CLI Addict
CLI Addict
Posts:245


07/12/2008 11:02 AM  
Are you operating against the mail server, or the desktop mail store?

Community Director, PowerShellCommunity.org
Co-host, PowerScripting Podcast (http://powerscripting.net)
Author, TechProsaic (http://halr9000.com)
glnsizeUser is Offline
Shell Enthusiast
Shell Enthusiast
Posts:60

07/12/2008 3:44 PM  

I used the following code to read my inbox.  more an exercise in "can I"... Having said that... Is this a onetime thing, or will this be a production resource?  Outlook is a cool little app, but processing mail from your local mail client.  Then using that data a populate a SQL database, bad juju...  Love to help all the same.

 

$outlook = New-Object -ComObject Outlook.Application
#you will have to grant permission within outlook. 
$mailbox = $outlook.session.folders | ? { $_.Name -eq "your Mailbox name"}

#lots of GM from here on out... very self explanatory
($inbox = $mailbox.Folders | ? { $_.name -eq "inbox"}).items | % { what-do-you-want}

 

-Glenn

 

gsxrian99User is Offline
New Member
New Member
Posts:7

07/13/2008 5:30 AM  

Desktop Mail Store.

But if there is a better way of doing it i'm all ears!

Thanks 

gsxrian99User is Offline
New Member
New Member
Posts:7

07/13/2008 5:37 AM  

This will be an ongoing situation.

At the moment i have to maually setup batch jobs each night to do this, and it is just waaay too much human involvement!

With exchange accepting emails automatically, i want to find a way to continue this process but instead of stopping at the inbox, i want to add one more step and take it into the sql DB.

I dont see why it should be that difficult, its all within the Microsoft family of products. 

Would exchange 2007 offer any advantages?

Why the bad juju?

Thanks 

halr9000User is Offline
CLI Addict
CLI Addict
Posts:245


07/13/2008 6:14 AM  
I believe what Glenn is getting at is that it's not good practice to rely on automating desktop applications in this way. A much better technique would be to operate against the server. But the server is already using a database, so that begs the question--what the heck are you doing, anyhow? Is this something that could be better accomplished by working with the emails at the server? Why does the data need to be copied into a second database?

See if you can answer these and maybe we can help you build a really robust solution. Relying on automating things on the client side is asking for trouble.

Community Director, PowerShellCommunity.org
Co-host, PowerScripting Podcast (http://powerscripting.net)
Author, TechProsaic (http://halr9000.com)
glnsizeUser is Offline
Shell Enthusiast
Shell Enthusiast
Posts:60

07/13/2008 4:28 PM  

Yup, dead on Hal, Like I said before, if you’re doing this for yourself… Let’s go, but I wouldn’t make it part of your infrastructure.   Not using Outlook, your pc blows up, or gets a Trojan and BAM. Bad Juju.

I still don’t know what you need to do… so it’s hard to give an informed answer, but if you’re exporting the data to SQL. I’m with Hal, either A. the data is in exchange leave it there, or B. the data shouldn’t be in exchange keep it in SQL. 

On the other hand, if you’re comfortable with it.. No a big deal.

I assume all these emails are going to a common folder. Let’s assume that folder is STUFF…

GLENN SIZEMORE
INBOX
     STUFF
          MORE_Stuff

The posh, to get at those messages would be.

 

$outlook = New-Object -ComObject Outlook.Application
#attach to the mail store
$mailbox = $outlook.session.folders | ? { $_.Name -eq "GLENN SIZEMORE"}

#very long hand!
$inbox = $mailbox.Folders | ? { $_.name -eq "inbox" }
$stuff = $inbox.Folders | ? { $_.name –eq "STUFF"}
$more_stuff = $stuff.Folders | ? {$_.name –eq "MORE_Stuff"}

# lets say you wanted to read every message in "Stuff"
# Any message with SQL in the subject. Mark it as read and move it to more_stuff..

# subject should read:
# USER: glennsizemore schedualed for SQL maint.

#scriptblock... to find userID
$Finduser = {$_.substring($_.indexof("USER:") + 6,$_.indexof(" schedualed") -6)}

$stuff.items | ? {$_.subject -match "SQL" } | % {
#mark as read
$_.UnRead = $FALSE

#extract userid from subject
$user = $_.subject | &$Finduser

#move the message
$_.move($more_stuff)
}

Do you have the SQL side already? You question was originaly about outlook.

 

~Glenn

gsxrian99User is Offline
New Member
New Member
Posts:7

07/13/2008 5:54 PM  

I guess it does not have to be done on the desktop.

I have emails forwarded from various resources across the US into a generic mailbox(one point of data import). Some times up to 1000, two to three page word, pdf, html or text documents per day, and this number may grow to 5k per day.

The information is imported into a sql DB for ease of searching the content via full text indexing.

I know you can search & index via outlook or the windows search feature but it is too slow.

Once the data is in sql, the employees can access the sql DB and do there own searching.

The sql DB offers a more centralized solution.

I currently have about 1 million records in the DB and could growto 10 million quite easily in the next couple of years.

Speed of data import is important as is speed of data access.  

Automation of the whole process is very important!

Thanks for the feedback:)

glnsizeUser is Offline
Shell Enthusiast
Shell Enthusiast
Posts:60

07/13/2008 6:17 PM  

I am not a DBA, not even close, but I know SQL can directly ingest that data. My suggestion would be to get Exchange out of the mix.  You sir need an application, not anything huge.  PHP or ASP would be my choice ingredient if your looking for homegrown, not Powershell.  You are exposing way too much power for a simple data ware house. 

It's one thing if SQL injection takes down the DB, but Powershell can turn it ALL OFF!

My 2c, focus on automating before it gets to exchange.  Perhaps look into a CMS, you have Windows Server standup SharePoint, or an open source variant like jumbia, or Mambo… Once the data hit’s exchange your already past automation.

~Glenn

halr9000User is Offline
CLI Addict
CLI Addict
Posts:245


07/13/2008 7:35 PM  
I agree with some of what Glenn is saying. Let's break this down:

- The scale of your needs is...rather large.
- Your data is going through email into Exchange--but is ultimately useless there.
- Your desired end state is in a SQL database.

Now let's talk about what you are doing now, or wanting to do anyway:

- Email is a slum where >90% of the messages hitting your edge servers is spam.
- Your Exchange infrastructure must serve the needs of employees at their desktops.
- As your data grows, that will increase the load on your Exchange service.
- As you don't need the data in Exchange, this will end up costing you money in upgrades for no reason.

I totally agree with Glenn that Exchange is just getting in the way here. If possible, I think you would want to take not just Exchange, but email out of the picture. That may or may not make sense depending on the remote sources of data. But I would seriously think about it.

That having been said, I don't see any inherent problems with using PowerShell to automate some of this. But I don't know where it makes sense to plug it in. What you need is a three-tier application built around this...Client -> middleware (queue / notification services, reporting etc.) -> database. If you have a budget then you may want to start getting some consulting quotes.

Community Director, PowerShellCommunity.org
Co-host, PowerScripting Podcast (http://powerscripting.net)
Author, TechProsaic (http://halr9000.com)
gsxrian99User is Offline
New Member
New Member
Posts:7

07/14/2008 4:56 AM  

Email is the easiest way for the data to get from mutliple remote locations, not always controlled by me, to a cental location that i do control. 

I can move it from email into same generic environment, but that involves one more step.

Oh the joy's of IT:)

I was told by a SQL Server DBA that SQL could import emails directly into the SQL SB.

He went awol, so i went to the SQL forums for advice, they sent me here, now i hear the SQL DBA reference again with data import directly into the DB:)       

It reminds me of a dog chasing its tail! 

halr9000User is Offline
CLI Addict
CLI Addict
Posts:245


07/14/2008 5:22 AM  
"they sent me here" -- I just read that post and that guy seems to be your problem, not your solution. :)

I did some searching and I think the right tool for the job is SSIS SQL Server Integration Services. I don't know a damn thing about it though.

Community Director, PowerShellCommunity.org
Co-host, PowerScripting Podcast (http://powerscripting.net)
Author, TechProsaic (http://halr9000.com)
gsxrian99User is Offline
New Member
New Member
Posts:7

07/14/2008 6:05 AM  

Thanks for your help!

I will keep digging:)

JaykulUser is Offline
Shell User
Shell User
Posts:32


07/15/2008 6:54 AM  
You can treat the Exchange server as a database using the Exchange OLE DB Provider (ExOLEDB) ( http://msdn.microsoft.com/en-us/library/aa125996.aspx & http://msdn.microsoft.com/en-us/library/aa564359.aspx) ... this includes running SQL queries to search and retrieve, etc.

So yes, you should be able use SSIS to move the data over.

Another option would be to use BizTalk, which is a general purpose data moving product that might help you get where you need to be faster -- but of course, it's not included in SQL Server ;)
gsxrian99User is Offline
New Member
New Member
Posts:7

07/16/2008 10:43 AM  
Thanks
You are not authorized to post a reply.
Forums > Using PowerShell > General PowerShell > Importing Emails into SQL Server DB



ActiveForums 3.7
right
   
footer Sponsored by Quest Software • SAPIEN Technologies • ShellTools, LLC • Microsoft Windows Server 2008 footer
footer