gsxrian99
 New Member Posts:7
 |
| 07/12/2008 7:01 AM |
|
I am looking to automate an import process using powershell.
My goal is to automatically have emails that come into my Outlook 2007 Inbox (XP) get imported into my SQL Server 2005 database. (Windows server 2003)
I also have windows server 2008 and sql server 2008 as an option, but am working thru some install issues with sql 2008 at the moment:(
Is Powershell the best way to do this?
Thanks
|
|
|
|
|
halr9000
 CLI Addict Posts:245

 |
| 07/12/2008 11:02 AM |
|
| Are you operating against the mail server, or the desktop mail store? |
|
Community Director, PowerShellCommunity.org Co-host, PowerScripting Podcast (http://powerscripting.net) Author, TechProsaic (http://halr9000.com)
|
|
|
glnsize
 Shell Enthusiast Posts:60
 |
| 07/12/2008 3:44 PM |
|
I used the following code to read my inbox. more an exercise in "can I"... Having said that... Is this a onetime thing, or will this be a production resource? Outlook is a cool little app, but processing mail from your local mail client. Then using that data a populate a SQL database, bad juju... Love to help all the same.
$outlook = New-Object -ComObject Outlook.Application
#you will have to grant permission within outlook. $mailbox = $outlook.session.folders | ? { $_.Name -eq "your Mailbox name"}
#lots of GM from here on out... very self explanatory ($inbox = $mailbox.Folders | ? { $_.name -eq "inbox"}).items | % { what-do-you-want}
-Glenn
|
|
|
|
|
gsxrian99
 New Member Posts:7
 |
| 07/13/2008 5:30 AM |
|
Desktop Mail Store.
But if there is a better way of doing it i'm all ears!
Thanks |
|
|
|
|
gsxrian99
 New Member Posts:7
 |
| 07/13/2008 5:37 AM |
|
This will be an ongoing situation.
At the moment i have to maually setup batch jobs each night to do this, and it is just waaay too much human involvement!
With exchange accepting emails automatically, i want to find a way to continue this process but instead of stopping at the inbox, i want to add one more step and take it into the sql DB.
I dont see why it should be that difficult, its all within the Microsoft family of products.
Would exchange 2007 offer any advantages?
Why the bad juju?
Thanks |
|
|
|
|
halr9000
 CLI Addict Posts:245

 |
| 07/13/2008 6:14 AM |
|
I believe what Glenn is getting at is that it's not good practice to rely on automating desktop applications in this way. A much better technique would be to operate against the server. But the server is already using a database, so that begs the question--what the heck are you doing, anyhow? Is this something that could be better accomplished by working with the emails at the server? Why does the data need to be copied into a second database? See if you can answer these and maybe we can help you build a really robust solution. Relying on automating things on the client side is asking for trouble. |
|
Community Director, PowerShellCommunity.org Co-host, PowerScripting Podcast (http://powerscripting.net) Author, TechProsaic (http://halr9000.com)
|
|
|
glnsize
 Shell Enthusiast Posts:60
 |
| 07/13/2008 4:28 PM |
|
Yup, dead on Hal, Like I said before, if you’re doing this for yourself… Let’s go, but I wouldn’t make it part of your infrastructure. Not using Outlook, your pc blows up, or gets a Trojan and BAM. Bad Juju.
I still don’t know what you need to do… so it’s hard to give an informed answer, but if you’re exporting the data to SQL. I’m with Hal, either A. the data is in exchange leave it there, or B. the data shouldn’t be in exchange keep it in SQL.
On the other hand, if you’re comfortable with it.. No a big deal.
I assume all these emails are going to a common folder. Let’s assume that folder is STUFF…
GLENN SIZEMORE
INBOX
STUFF
MORE_Stuff
The posh, to get at those messages would be.
$outlook = New-Object -ComObject Outlook.Application #attach to the mail store $mailbox = $outlook.session.folders | ? { $_.Name -eq "GLENN SIZEMORE"}
#very long hand! $inbox = $mailbox.Folders | ? { $_.name -eq "inbox" } $stuff = $inbox.Folders | ? { $_.name –eq "STUFF"} $more_stuff = $stuff.Folders | ? {$_.name –eq "MORE_Stuff"}
# lets say you wanted to read every message in "Stuff" # Any message with SQL in the subject. Mark it as read and move it to more_stuff..
# subject should read: # USER: glennsizemore schedualed for SQL maint.
#scriptblock... to find userID $Finduser = {$_.substring($_.indexof("USER:") + 6,$_.indexof(" schedualed") -6)}
$stuff.items | ? {$_.subject -match "SQL" } | % { #mark as read $_.UnRead = $FALSE #extract userid from subject $user = $_.subject | &$Finduser #move the message $_.move($more_stuff) }
Do you have the SQL side already? You question was originaly about outlook.
~Glenn |
|
|
|
|
gsxrian99
 New Member Posts:7
 |
| 07/13/2008 5:54 PM |
|
I guess it does not have to be done on the desktop.
I have emails forwarded from various resources across the US into a generic mailbox(one point of data import). Some times up to 1000, two to three page word, pdf, html or text documents per day, and this number may grow to 5k per day.
The information is imported into a sql DB for ease of searching the content via full text indexing.
I know you can search & index via outlook or the windows search feature but it is too slow.
Once the data is in sql, the employees can access the sql DB and do there own searching.
The sql DB offers a more centralized solution.
I currently have about 1 million records in the DB and could growto 10 million quite easily in the next couple of years.
Speed of data import is important as is speed of data access.
Automation of the whole process is very important!
Thanks for the feedback:) |
|
|
|
|
glnsize
 Shell Enthusiast Posts:60
 |
| 07/13/2008 6:17 PM |
|
I am not a DBA, not even close, but I know SQL can directly ingest that data. My suggestion would be to get Exchange out of the mix. You sir need an application, not anything huge. PHP or ASP would be my choice ingredient if your looking for homegrown, not Powershell. You are exposing way too much power for a simple data ware house.
It's one thing if SQL injection takes down the DB, but Powershell can turn it ALL OFF!
My 2c, focus on automating before it gets to exchange. Perhaps look into a CMS, you have Windows Server standup SharePoint, or an open source variant like jumbia, or Mambo… Once the data hit’s exchange your already past automation.
~Glenn |
|
|
|
|
halr9000
 CLI Addict Posts:245

 |
| 07/13/2008 7:35 PM |
|
I agree with some of what Glenn is saying. Let's break this down: - The scale of your needs is...rather large. - Your data is going through email into Exchange--but is ultimately useless there. - Your desired end state is in a SQL database. Now let's talk about what you are doing now, or wanting to do anyway: - Email is a slum where >90% of the messages hitting your edge servers is spam. - Your Exchange infrastructure must serve the needs of employees at their desktops. - As your data grows, that will increase the load on your Exchange service. - As you don't need the data in Exchange, this will end up costing you money in upgrades for no reason. I totally agree with Glenn that Exchange is just getting in the way here. If possible, I think you would want to take not just Exchange, but email out of the picture. That may or may not make sense depending on the remote sources of data. But I would seriously think about it. That having been said, I don't see any inherent problems with using PowerShell to automate some of this. But I don't know where it makes sense to plug it in. What you need is a three-tier application built around this...Client -> middleware (queue / notification services, reporting etc.) -> database. If you have a budget then you may want to start getting some consulting quotes. |
|
Community Director, PowerShellCommunity.org Co-host, PowerScripting Podcast (http://powerscripting.net) Author, TechProsaic (http://halr9000.com)
|
|
|
gsxrian99
 New Member Posts:7
 |
| 07/14/2008 4:56 AM |
|
Email is the easiest way for the data to get from mutliple remote locations, not always controlled by me, to a cental location that i do control.
I can move it from email into same generic environment, but that involves one more step.
Oh the joy's of IT:)
I was told by a SQL Server DBA that SQL could import emails directly into the SQL SB.
He went awol, so i went to the SQL forums for advice, they sent me here, now i hear the SQL DBA reference again with data import directly into the DB:)
It reminds me of a dog chasing its tail! |
|
|
|
|
halr9000
 CLI Addict Posts:245

 |
| 07/14/2008 5:22 AM |
|
"they sent me here" -- I just read that post and that guy seems to be your problem, not your solution. :) I did some searching and I think the right tool for the job is SSIS SQL Server Integration Services. I don't know a damn thing about it though. |
|
Community Director, PowerShellCommunity.org Co-host, PowerScripting Podcast (http://powerscripting.net) Author, TechProsaic (http://halr9000.com)
|
|
|
gsxrian99
 New Member Posts:7
 |
| 07/14/2008 6:05 AM |
|
Thanks for your help!
I will keep digging:) |
|
|
|
|
Jaykul
 Shell User Posts:32

 |
| 07/15/2008 6:54 AM |
|
You can treat the Exchange server as a database using the Exchange OLE DB Provider (ExOLEDB) ( http://msdn.microsoft.com/en-us/library/aa125996.aspx & http://msdn.microsoft.com/en-us/library/aa564359.aspx) ... this includes running SQL queries to search and retrieve, etc. So yes, you should be able use SSIS to move the data over. Another option would be to use BizTalk, which is a general purpose data moving product that might help you get where you need to be faster -- but of course, it's not included in SQL Server ;) |
|
|
|
|
gsxrian99
 New Member Posts:7
 |
| 07/16/2008 10:43 AM |
|
| Thanks |
|
|
|
|