Acabo de escribir como la dedicación del equipo de desarrollo fue el clave a nuestro éxito. Aparte de los ingenieros a Microsoft trabajamos con Southworks que es un grupo de gente muy apasionante de su trabajo y de tecnología.

Este proyecto requerido mucha dedicación porque no teníamos mucho tiempo y teníamos que desarrollar mucha tecnología. La mayoría del equipo estaba en Buenos Aires pero esto no fue un problema porque manteníamos comunicación muy abierta y tenían líderes muy efectivos. La verdad es que el equipo trabó en una fiesta porque teníamos que presentar el demo al equipo de BillG el día siguiente. Muchas gracias a las familias de ellos por tener la paciencia.

A mi gente de Southworks… Les saludo y tienen el respecto de muchos en Redmond!

There were a lot of people involved with pulling together the Data Platform Demo for the TechEd Keynote.  I want to take a minute to quickly thank the Southworks team who’s dedication really made the difference.

Special thanks to Tim Osborn, Pablo “Lito” Damiani and Johnny Halife.

One of the biggest challenges we faced when designing this demo was trying to make the breadth of Microsoft’s Data Platform technologies easily understandable in 5 minutes.  The major architecture components are SQL Server 2008, SQL Server Data Services (SSDS), SQL Server Compact and Sync; which I outlined in a previous post.  We’ve been following the blogosphere as closely as possible and, understandably so, one of the areas we’ve seen a bit of confusion around is the type support and conversion between the Entity Data Model, SSDS and SQL Server 2008.

There are a lot of moving parts here so I’ll do my best at explaining how everything was integrated and how we got around some of the differences in type support of the data platform technologies.

Web Application

Bloggers use this application to submit geo-tagged articles and images which are stored in SSDS.  We embedded a Virtual Earth control which the users use to manually geo-tag their content.  Those who have been following SSDS closely may have noted that SSDS does not currently support blob storage or spatial types. 

• Blob Storage in SSDS - This is the number one feature that customers are asking for right now with SSDS.  Until SSDS supports blobs our current workaround was to Base64 Encode the images for storage in SSDS.
• Spatial Storage in SSDS - Currently SSDS does not have support for Spatial types.  This was an easy work around for us by converting the spatial POINT coordinates to the Well Known Text (WKT) format and storing them as text. 

Mobile Application

In this scenario we are also using a Windows Mobile device that allows bloggers to submit photos which are automatically sent to SSDS via the Microsoft Sync Framework.  Here are some key points:

• A SQL Server Compact database stores the application’s meta-data
• Images are stored on the device’s file system
• Geo-tagging is pulled directly from the GPS enabled device.  The app caches the last known GPS coordinates in case of lack of GPS connectivity.
• The Sync provider running on the device converts the geo-data to WKT and the image to Base64 text as in the web app

Sync from SSDS to SQL Server

Our application uses a powerful WPF desktop application connected to a local SQL Server 2008 database.  Since SQL Server 2008 supports Spatial we have the ability to run high performance spatial queries which aren’t currently possible in SSDS.  FILESTREAM allows us to use the high-performance of the file system for binary file storage along with transactional consistency and great manageability of the database.  Type conversion here was very simple and outlined below:

• Sync and FILESTREAM - The Sync Provider sitting on our SQL Server database pulls the Base64 binary data from SSDS and inserts it into a varbinary(max) FILESTREAM column in our database.
• Sync and Spatial - This Sync Provider inserts the WKT POINT data into a SQL Server GEOGRAPHY type.

EDM, Spatial and FILESTREAM

  Our WPF application uses the Entity Data Model (EDM) to provide the application developers with a more natural business representation of the data. This allows the database model and application data model to evolve independently. Currently the EDM has limited support for FILESTREAM and does not natively support the new SQL Server 2008 Spatial types (GEOGRAPHY and GEOMETRY).  These were also very easy to work around in the following ways.

• EDM and FILESTREAM - Currently the EDM treats FILESTREAM in the same way it treats any other varbinary(max) column.  You get the transaction consistency and manageability of the database.  It interacts with the FILESTREAM data through T-SQL so you don’t get the Win32 streaming performance that FILESTREAM has the ability to provide.  If that type of read/write performance is needed you can easily write a section of code that interacts with traditional database connections and SQL.
• EDM and Spatial - Currently the EDM does not support the new Spatial types.   Our WPF application had two main requirements for Spatial: high performance queries and showing the Spatial meta-data.  We wrote a stored procedure for the queries and mapped a calculated column which converted the spatial data into WKT for displaying the meta-data.

Summary

There were definitely some things that we had to consider when building this application but in the end none were major barriers.  The application works great and was an enjoyable development experience. 

Just minutes ago at TechEd 2008 Developers in Orlando Dave Campbell, Technical Fellow, was on stage with Bill Gates doing the Data Platform demo which included a host of exciting Microsoft Technologies.

• SQL Server 2008 - Supports any data
• SQL Server Data Services (SSDS) - Quickly provisioning for unpredictable scale
• Microsoft Sync Framework - Keeping all data synchronized

Building this demo has been one of the most rewarding experiences in my career.  Everyone involved has put in 110%.

When we set out to build this demo we spent a lot of time making sure that the architecture was something that we have heard customers asking for.  Since SSDS is such a new piece of the Microsoft Data Platform going through this process was an interesting experience.

In this article I will quickly highlight the architecture of this application and describe why we made some of the architecture decisions.

Scenario

This application was built around a fictitious company called Trey Research.  Trey Research is a news agency that has launched a new strategy aimed at turning bloggers into paid journalists by paying them for their articles and photos.  Bloggers from around the world submit articles and images through either a web application or Windows Mobile app.  New analysts at Trey Research find the best articles and images for a given geographic area of interest, combine them into a story and sell them to companies like MSNBC; paying the content creators in the process.

Architecture

Here is a high-level overview of the architecture.

SQL Server 2008 and WPF

So starting from the News Analyst WPF App and SQL Server 2008.  We chose to use local SQL Server database connected to a WPF desktop application for a number of reasons including.

• High Performance of a Local Database
• Powerful Analytics of the SQL Server platform
• Ability to execute Spatial Queries to search for relevant content
• Storage of all content types including geo-tagged text and images

News Analysts at Trey Research can search for content with a power UI including Spatial and Time based queries.  SQL Server uses powerful analytics on the back end to determine the target demographic for the content.  Some of the exciting technologies being used are:

• FILESTREAM to store binary data
• New DateTimeOffset type used to store the date and time a photo was taken, which preserves time zone information
• Spatial queries and indexes which allow for the fast retrieval of geo-tagged data.
• SQL Server Reporting Services to provide rich visualization of analytical data.

SQL Server Data Services, Web Application and Mobile Device

All of this content is submitted via a web site or via a Windows Mobile application.  The content submitted via the web site or mobile app are stored in SSDS.  Their core competency is managing news content, not building Internet scale data centers.  Trey Research decided to use SSDS as the data storage platform for a number of reasons:

• To quickly provision for the unpredictable scale of what is turning out to be a very popular web site
• The nature of news is that a large event can produce huge spikes in traffic, so they are relying on Microsoft’s global infrastructure and Service Level Agreements
• SSDS is acting as a Data Hub where web and mobile devices submit content

Microsoft Sync Framework

The final piece of magic in this application is the Microsoft Sync Framework.  Sync is used to pull data down from SSDS into the local SQL Server database in addition to moving data from the mobile device into SSDS.  The sync framework provides a powerful platform for dealing with things like conflict detection, etc.

Summary

As you can see there were a number of architecture decisions that had to be made.  Trey Research is using many of the powerful features of the Microsoft Data Platform to quickly build this application.  I’ll continue to post information about this demo.

The speaker gave two examples which I thought explained her point very well.

Example 1
The Chinese professor may be hot and want the window opened.  Instead of asking her student to open the window she may ask the student, "Are you hot?"   Hopefully the student understands that what she was really asking in an indirect way was, "Will you please open the window?".

Example 2
An American visitor goes to a Chinese coworker’s house for dinner.  The American guest compliments what a beautiful painting that’s on the wall.  The next day the American finds the painting on her desk.  The Chinese visitor thought that the American’s compliment was a subtle request for the painting.

That’s exactly the type of critical understanding of a different culture which I think helps get along in new environments.

Has your experience in China been the same? 

Today, I’m at an internal Microsoft event related to doing business in China.  The session I’m looking forward to the most is related to the cultural aspects of working with Chinese partners.  Anytime that I have the opportunity to travel to another country I always buy the Culture Smart! travel book.  In my experience knowing the culture is the most important part of travel.

Is it Windows Live Sky Drive?  Is it Astoria?

The answer is no…  It’s neither of the two above although it does have some features of each of them.

The best way to get the low-down on SSDS to watch Nigel Ellis’ presentation from MIX. 

I’d also recommend subscribing to the SSDS team blog.

The links above are great resources for starting to go deep…  Let’s see if I can sum up SSDS in an extended elevator pitch.

SSDS is Microsoft’s cloud storage offering built on top of SQL Server.  It’s a data storage service backed by an enterprise class SLA accessible via REST and SOAP APIs.  It’s currently accessible via an invitation beta program, a beta version will be broadly available mid-summer and is targeted to Go Live in H1 of ‘09.

So what is this new cloud storage offering by Microsoft anyway?

• Microsoft hosted data storage service based on SQL Server
• Accessible via REST and SOAP web services APIs

It’s an exciting addition to the Microsoft line up of data storage products and services.  It’s great to finally be able to talk about it and we are really excited to see how developers use the service.

I’ll continue to blog about this as well as post some samples.

For years we’ve been told that large binary files should never be stored in the database…  Are you telling us to start storing these files in the database now?  If so, why?  Are we just supposed to throw this best practice out the window?

This is really a great question which has prompted some interesting discussions.  Obviously the answer is not black and white. So let’s start by looking at what has changed.

SQL Server 2008 is now a very powerful engine for storing binary files. 

• These files can be accessed through high performance Win32 streaming API’s in addition to T-SQL.
• These files are managed by SQL Server in their own file groups which can be backed up restored along with the rest of your SQL Server data.  On the flip side you aren’t required to backup and restore these file groups.
• Reading and writing these files can now be part of a database transaction.

So you might be thinking to yourself…

Sounds great!!!  Let’s start storing all of our binary data in SQL Server.

Well, there are some considerations to be made before signing up to rewrite your app to take advantage of FILESTREAM.  Here are some of the main considerations.

Do other applications need direct access to your binary files?

If you read my article about writing files to FILESTREAM you probably noticed that you have to go through SQL Server to access the data in FILESTREAM.  There is no concept of
OpenFile(”C:\Path_To_My_File\File_Name.docx”)

Does your architecture require database mirroring?

Database mirroring does not yet support FILESTREAM.

Those are just a couple of the things to think about.  I’d recommend checking out our FILESTREAM sample on CodePlex and make some decisions for yourself. 

FILESTREAM is a great technology and we are really excited to see how developers incorporate it into their applications.  Feel free to post comments here about your experience integrating FILESTREAM into your architecture.