Never Talk When You Can Nod

Externalizing BLOB Storage in SharePoint 2010

OK, so this entry should be called SharePoint Conference Day Three but to be honest I only attended one session – the rest of the day was taken up with meetings with Microsoft and Partners.

So as a rather sad indictment of the state of my life, the best session of the week for me was "Externalizing BLOB Storage in SharePoint 2010". I was absolutely enthralled with this session. Srini Acharya and Burzin Patel from the Microsoft SQL server advisory team did a very nice job of addressing the concept of RBS and how it fits in to the new SharePoint 2010 architecture.

They started with an overview of the general RBS concepts and also a demo of RBS using Microsoft’s own free file stream provider that just dumps the BLOBs on to a local file server. To be clear, RBS on its own without a provider will not do anything, Microsoft have the aforementioned file stream provider and vendors are being encouraged to develop their own. During the Q&A session they made the point that customers should not write their own providers and that vendors should work with Microsoft on their offerings.

Burzin mentioned that "BLOBs typically account for 60-70% of total content" in the SharePoint content database. This is a lower number than I’ve heard quoted previously although most of the numbers you see quoted are from vendors trying to sell externalization solutions so maybe they cannot be trusted ;-)

EBS vs. RBS

It seems obvious that the team have been making an effort to bring SharePoint awareness in to the RBS layer. They have added an ‘RBS Maintainer’ to sync certain operations between SharePoint and RBS. I think that the name of this component needs some Microsoft attention BTW.

They presented a table outlining the key advantages of RBS over EBS.

• RBS has Managed interface whereas EBS’s is unmanaged.
• The RBS BLOB store scope is per Farm, EBS is per Content DB.
• SharePoint 2010 adds the Configurable Maintainer to sync operations from SharePoint down to RBS.
• RBS also has a PowerShell-based UI within SharePoint, this was pretty rudimentary but again it shows a commitment to the integration.
• RBS supports multiple providers. I think that you will be able to register multiple providers for a farm and then select which ones are used and when – I don;t understand the scope of that decision though, I’d love to be able to enable a provider for site A and a different one for site B, (or even by document library perhaps).
• Best of all…the RBS implementation includes PowerShell-based migration between content stores. Translation? You can move BLOBs from SQL to any RBS provider, from RBS provider to RBS provider and from EBS to RBS… They are looking at a ‘shallow migration’ where the BLOB would stay where it is but the management of it would be migrated (only for already externalized BLOBs). This is a huge thing – one of the senior engineering leads was sitting by me during the session and when this was announced he turned around and gave me a huge grin…given that I’d met with him to discuss RBS earlier in the day it looks like he’d intentionally kept this nugget from me…you know who you are and I will get revenge!!

Issues According to Microsoft

I have to take care when talking about the architectural challenges with storing BLOBs in SQL because I am not in the business of criticizing SharePoint (anymore) so the next section was a blessing for me. So here it is…Microsoft’s assessment of the issues with storing BLOBs in SQL…not mine, Microsoft’s…taken verbatim from their slide!!

• Optimal Capital Expense
  • Trade cost effective BLOB storage for expensive SQL Storage
  • Ability to group/store BLOB separate from Metadata
• Optimal Operational Expense
  • Storage management beyond SQL
  • Facilitates cost effective/optimal backup/DR policies for BLOB vs. Metadata
• Take advantage of advanced storage features provided by BLOB store vendors
  • Expunge, multiple storage locations, immutable writes
  • Guaranteed retention, guaranteed deletion,
• Hierarchical Storage Management
  • SQL and layers of BLOB stores offer more savings in CapEx and OpEx
  • Efficient access patterns.

That’s Microsoft’s list – unedited. 

They then did a re-cap of where we are with RBS and SharePoint.

• RBS still needs a storage vendor to supply a provider and a BLOB store. OOTB RBS doesn’t do anything without a provider.
• RBS includes a RBS Maintainer and a Provider Library for SharePoint 2010. The maintainer synchronizes BLOB delete/orphan operations between SharePoint and RBS. The library is just that, a list of registered providers.
• All BLOBs in an active content DB will be externalized – the level of selectivity is still binary i.e. all or nothing.

How does it work? Defines and exposes 3 views for interaction"

• Application View
  • Interacts with the SharePoint WFE, Provider Library, SQL DB
  • Implemented by SharePoint 2010 - transparent to the user
• Administrator View
  • PowerShell commandlets - calls set of SPs, function as in SQL
  • Installation, configuration, provisioning, RBS maintainer etc.
• Provider View
  • Defines an interface that should be implemented but the BLOB storage provider.

The jumped in to a really nicely done demo

• Went to a RBS enabled site
• Opened a PowerShell window
• Showed that the file stream provider was disabled
• Showed a SQL select on the WSS_Content table. There were 73 results but they all had RbsId of null so none of the objects were externalized.
• Imported an image
• Re-ran the query and saw 74 rows – still nothing with a RbsId.
• Enabled the RBS provider
• Added another image
• Re-ran the query
• 75 rows but one of the rows now has an RbsId
• Went to the local file system on the C drive
• Document was on the file system with an object name of the GUID (he opened it to show that it was the same file).

It didn’t look like much but for those customers struggling with SQL bloat this was manna from heaven.

Next they went on to describe the RBS Maintainer module. This is the piece that creates an element of awareness between SharePoint and RBS. Here are the key points:

• It tracks the deletions in SharePoint and pushes them to the BLOB store. Interestingly it includes the ability to set up rules to delay the delete from the BLOB store to make it easier to manage restores – if the BLOB is still in the external store when you restore SQL then you are probably in good shape.
• Does the clean-up of BLOBs for failed transactions - orphaned BLOBs. This and the line above are basically garbage collection management.
• Can be on the DB server or separate box
• Can be scheduled to run periodically – I guess it is a batch job rather than an inline process.

They went on to discuss How to Backup works. I think they oversimplified this section – my advice would be to think carefully about your chosen backup method. Here was their advice:

• Backup:
  • Start SQL Backup first
  • Then backup BLOB store
• Restore
  • First restore BLOBs
  • Then Restore SQL

The team showed some performance data that I found questionable – it showed that externalization of data did not have a negative effect on the performance of SPP. I suspect that this was all they wanted to show but IMHO their data sample contained files that were too small and I’m guessing there were not suffering from the fragmentation issues that I mentioned in my earlier Blog post. I’m confident that if they used a more realistic sample they’d see an improvement in performance (reads and writes) especially directly to an equivalent file system.

There were a large number of follow-up questions in the session. Many of them were about whether RBS would make it easier to get existing file system data in to SharePoint… I’ll reserve my comments on that for a later entry I think.

Again, Srini and Burzin did a stellar job explaining this abstract concept to a packed audience.

SharePoint Conference Day Two

Today was a bit more fragmented because I had some Microsoft meetings and some EMC partner meetings but I got to some good sessions regardless…

Attended the EMC consulting best practices session "EMC: Good, better, best: Understanding Enterprise SharePoint Deployment Best Practices". The EMC team acted out a scenario that represented a typical customer/consulting engagement. This worked well and the audience seemed engaged as they moved through their demo. The size of the crowd suggests that there are a lot of people out there who are looking for more information and best practices. Not a surprise given the changes in 2010.

My next session was Revolutionizing the SAP user experience… with SharePoint. Truthfully, most of this session was way above my head as I am new to the SAP world but the rest of the audience seemed to be enthralled! The heart of the session seemed to be that if you've chosen to use SAP then SharePoint has a place in exposing the SAP systems to end users. They showed examples of taking 7 SAP screens and creating a single SharePoint-based view with only a subset of data available.

The most interesting session of the day for sure was the Document Management Deep Dive with Ryan Duguid. The following are notes from that session...

High-level Messages

• SharePoint is all about "Traditional ECM" and collaboration brought together.
• Primary tag line - "ECM for the Masses"

Ryan showed a graph that shows how Traditional ECM is typically aimed at small number of types of content in a typical organization but not aimed at the larger number of non-controlled types of content. SharePoint is focused on that latter class of content types and on the breadth of content.

A bit of random marketing in the deck...

ECM Tenets: 3 'E's

• Easy to Use
• Everyone Participates
• Enterprise Ready

The Importance of Disposition

Ryan spent a considerable length of time talking about disposition. FWIW I agree entirely with this position. He had a graph that showed that starting today, with a 40% annual increase in data you will have 45TB of content in 10 years. If you dispose of just 10% of content each year you'll drop that to 25TB…dispose of 20% per year and it goes to 10TB…30% gets you 4TB. This shows how important it is to have a real disposition policy.

What's added in 2010 for ECM?

• They added unique document IDs to all objects - this seems like such a fundamental component that I do not understand how they managed previously without it - maybe it was hidden. The doc ID is optional in a site, they can be reset across all the site which is very odd and you can provide a prefix to ensure uniqueness, (which suggests that maybe they are only unique within a site or perhaps farm - this needs checking out).
• Ratings (click on a star - 1 through 5 to say how lovely something is) on any object in the site is being sold as a a big deal although this is something a bit alien to me. I could see it being popular in many cases…I'd just never given it any thought. The more I think about it the more I like this concept…
• Tagging - this one I get. Being able to create tags, apply them to an object and manage them. This is how navigation is going to be driven moving forwards. The model that I see SharePoint moving towards is one where the physical location of an object is there for the application of policies and security but not for navigation. You can optionally include the folder location as a navigation tag so it can also be a navigation factor but that’s incidental in this model.
• Drag and drop from Windows explorer into the multi file upload component – useful.
• Multi-select operations – again, great to see this and hard to imagine how SharePoint got away without having it.
• The ability to view compliance details of an object from the drop down menu. See if there's any retention, disposition, etc. applied to that document. Nice feature, very useful for admins, records managers and end users.
• Send To - used to be able to send to one record center, now you can reroute to multiple record centers or even just to another site, (I believe that all the functions previously associated with record centers can now be applied to any site). Ryan showed how you would set up a new record center. Looks like the decision to Copy, Move or Move and leave Stub seems to be per record center.
• Can apply retention policies to folders not just content types.
  • Within a folder you can define whether the objects declared as records are treated differently then other objects in the folder. This explains how it is they can simply select an object and click 'declare as record' without answering any other questions. The rules to determine the behavior once something is tagged as a record are defined in this way.

• Folders are a way of applying security, retention, disposition rules, etc on a location centric basis. I suspect that Microsoft are going to push faceted search/navigation as the way forward and folders as being a back end mechanism. This makes a lot of sense so long as the tagging is done correctly…a big risk perhaps?

Managed Metadata Service

• Supports type-ahead when adding a tag to an item - this is fully qualified so if you type a term that exists in many places in the taxonomy it will allow you to see the one that is in the right location.
• Want to support a folksonomy maturing in to the managed taxonomy. For example, you can send a message to the taxonomy owner to suggest new terms all from within SharePoint 2010
• Ryan showed how easy it was to create new taxonomy values…of course creating an agreed-upon taxonomy in the first place is the real challenge!!
• Note that the Managed Metadata Service can be shared across sites – this is a huge deal.
• It also supports multi language support so you can switch to a new language and see localized tags, (they don’t translate terms but allow you to enter multilingual tags.
• The hierarchy is "Groups, term sets, terms" this allows you to set rules across the hierarchy including ownership of the terms.
• Seems like there's been a lot of thought put in to this feature. Especially around making it easy to move things around and turn on/off the individual terms.

Navigation by metadata is probably the most significant feature that I've seen in a long time. I was going to say 'if it works' but if data continues to grow as predicted then maybe it has to.

One very important point is that Microsoft believe that in order to get users to invest in the effort of tagging content in the first place then they need to see a 3x return on that investment. Faceted navigation is that return…spend the time tagging today and find the content in whatever way you prefer tomorrow.

Document Sets

• Like an intelligent folder, they wanted it to look familiar so they make it look like a folder
• The set can have a landing screen which looks like a wiki - more than just a relationship object, it allows for collaboration around a collection of documents.
• It looks like a document can only live in one Document Set which might be a real problem. This needs validating.
• Ryan described it as a "Foundation for document assembly"

Content Type Syndication

Share objects, metadata, workflow, between farms.

Ryan demoed this - He opened the admin screen and defined where the content type hub was. He went to that site and provisioned a new content type just as you would in any farm/site. He then managed the publishing for that content type. He published it, waited for a sync job to run and then went back to a remote site where the new content type was available to be provisioned from the hub for local use.

Content Organizer

In site settings there are rules that can route content to specific locations. He set up a new rule that says for a specific content type for content that has a specific metadata value it would be moved to one location. It can provision a new folder based on the value of a specific piece of metadata. For compliance this would be a very compelling feature – especially if you can get metadata tagging done accurately at the right time.

SharePoint Conference Day One

I'm at the SharePoint 2009 conference in Vegas this week. Here are some notes from today – no conclusions, just some notes…I'll try to post something later in the week and I'll continue to tweet all week - @chapmaa

The Keynote

You've no idea how many bad keynotes that I've sat through in my life but today's was not one of them. The keynotes ran for 2:45 without a break but the presenters did such a good job that it did not feel long at all. So how did they do that? They had some passionate people on the stage, (most notable was Tom Rizzo), and they told the message in a way that the audience wanted to hear it not in the way that marketing wanted to present it. Within the first 30 minutes Rizzo was writing code in a live demo! They didn't just say what 2010 will do they showed what it will do and how. Without exception the team did an exceptional job.

As a side note, Jeff Teper called SharePoint "the ultimate Swiss army knife"…I think that’s a good description.

The recurring themes…

• Reduce costs…not sure how many times Ballmer mentioned reducing costs but it was a core message. Have fewer systems, fewer vendors, be more nimble. "Get more done with less" was the overriding message here.
• The cloud…they kept mentioning the cloud but they didn't give to much detail. This was perhaps the one area where it was higher-level than I'd have liked to have seen. Microsoft online was mentioned, the new web-based office tools were used heavily but no specifics about the actual cloud structure per se.
• Support for inter- and intranet support as well as support for the office online tools. These are two very different things but they got a little merged in the keynote.
• Streamlined deployment and debugging…a bit techie for me this but I could see the value of being able to easily deploy solutions.

Post keynote I attended intro sessions and ECM sessions…

The Content Management Messages

I'd say that in general ECM is being played down as just part of a much bigger play. For example, Ryan Duguid pointed out that you can tag, annotate and place legal holds on anything in SharePoint, not just documents. I was watching out for ECM comments for obvious reasons, here are some highlights:

• Overall ECM 2010 Changes
  • Scale ECM up and out - a general message and pretty vague
  • Taxonomy, workflow and document sets have been added
  • DAM (later qualified by Ryan as being not being full blown DAM but being media management for images and video including intelligent streaming direct from SharePoint)
  • More flexible governance (policy and RM support)
    • Declare as record - I need to talk to Ryan about this one because it looked like it just 'made the object immutable'  not actually declared it as a record
    • Manage records in place
    • Route to records center and leave a stub behind
      • I loath and detest this as an approach but Ryan addressed this by saying that Microsoft were giving customers a choice - IMHO this means that they can do it properly or they can stub ;-)
  • Increase in number of objects that can be managed by SharePoint
  • Master metadata management (see below - this one deserves its own section)
  • Document Set support…create a relationship between documents and manage them as a single document.
  • Offline support - (see below - this gets its own notes too)
  • Extension of routing table usage
    • Can be used anywhere on any content type and across sites
    • Can create a 'content organizer' as a drop zone
      • Users import content in to the content organizer and the rules will move the content in to the correct location. This seems to include rules that will look at the contents of the object, (Ryan showed moving anything with a social security number to a specific location).

Metadata Driven Navigation

SharePoint 2010 has a lot of focus on metadata taxonomy management. This still looks like it is not entirely fully baked yet but the basics are there and it is flexible. It looks like it scales across SharePoint sites/farms but also allows external systems to access the taxonomy. This is a good strategy for a few reasons:

• Strategically, if you own a company's metadata you become the information hub for all the information-centric systems
• If you can get users to tag content correctly at a logical time then you can remove the workload from them later and use rules to apply complex operations on the content further on in the information lifecycle.
• Most importantly navigating content based on metadata is much more natural, (you search and then activate filters to remove noise - think about how eBay let's you filter in real time).
• If the only thing that SharePoint 2010 brought was this faceted navigation then it would be worth the effort to upgrade for that alone – IMHO.

Offline Support

Microsoft talked about "SharePoint Workspace" a lot. This allows you to synchronize content from SharePoint to your desktop to:

1. Work offline
2. Use this as a way of working in low-bandwidth locations

The second use case above is a very interesting usable model and in some ways addresses one of SharePoint's enterprise limitations.

I’ll make an effort to learn more about this because it is potentially very interesting as a Microsoft partner.

Managing content on the file system

This is another one where I need to do some extra research but in a nutshell the file system has the capability to store metadata natively and then content can be pushed up in to a specific SharePoint site based on rules. As the content hits the SharePoint site the rules in a content organizer can further file away the content. It can leave stubs behind BTW, I’ll reserve my comments until I see how good the native file system is at masking these as being virtualized.

Here are my raw notes from Ryan’s talk – as I say, I’ll be working through this one a bit this week.

• File server resource manager. Windows Server 2008 R2
  • Apply metadata properties to content on the file server
  • Can have rules running on the file server
  • Can look inside content for patterns too.
  • Can auto upload content from the file server to SharePoint automatically.
  • They leave links behind that point to the SharePoint location.
    • On ingestion content is further redirected to a SharePoint folder location

 

I’d say that day was was very good – 7,400 attendees, almost perfect organization and a great vibe.

SharePoint Conference and New EMC Products

Next week is the annual SharePoint love fest – this year it is in Vegas. Unfortunately, although I’ll be at the conference my understanding is that I will not be able to Blog or Twitter on what I see/hear…something to do with ‘What happens in Vegas, stays in Vegas”. Seems like it makes the entire trip a bit of a waste of time ;-)

Seriously though, I will try to post some highlights although they may be Twitters rather than full blown blog entries. It looks like a huge proportion of the week is given over to SharePoint 2010 which is not great surprise I guess. You can follow me @chapmaa if you have nothing better to do.

The big news for me next week also includes the release of two new SharePoint-related products. I don’t normally Blog about my role at EMC and our specific technologies but this is a particularly big event for us and not surprisingly the products deal with some of the specific ‘opportunities for excellence’ that I’ve discussed in my Blog.

For those of you that are interested, here’s what we will be announcing at the conference:

• ‘My Documentum for Microsoft SharePoint’ is a SharePoint emulator for Documentum users. It allows end users to interact with content in a Documentum system without leaving the familiar look-and-feel of SharePoint. The key feature of this solution is that it is very faithful to the native SharePoint look and feel…in fact it is about as close as we can make it without breaching any legal agreements that we have with our friends in Redmond.
• Slightly more interesting from a technology perspective is ‘Documentum Repository Services for Microsoft SharePoint’[1]. This product makes use of EBS and will support RBS moving forward. The solution makes Documentum into a back-end storage and ECM system for SharePoint. This is especially useful for companies that want SharePoint’s ECM to play nicely with their broader corporate ECM backbone. It is a solution that leverages the concepts that I’ve rambled on about for the last 18 months or so – creating a model where content is stored in a back-end repository rather than SQL Server but in such a way that it is 100% transparent to SharePoint. Apart from just being damn cool this solution really unites two core enterprise solutions in a way that takes no value away from either system. I’m very proud of the fact that we developed the entire product in 13 months which is amazing for a full-blown enterprise strength solution. It has been in limited release for 6 months and is now available to all and sundry.

If you care you can come to the EMC booth and see more…I’m attending the conference as an attendee and will be attending sessions but we will have people much more personable and presentable manning the booth.

If you’ll be there then have fun, if you are not there then congratulations on having a life.

---

[1] You have to include ‘Microsoft’ before SharePoint in product names making this a sweet 55 characters long!

When to Externalize BLOBs…

I just re-read a really interesting technical report from the Microsoft research team that relates to when it is ‘better’ to store BLOBs in SQL vs. externalizing them on to a file system. The report is very well written and is worth plowing through because there’s a lot of good information hidden in there. You can find the source article at http://research.microsoft.com/apps/pubs/default.aspx?id=64525 .

Here’s my brief assessment of the take-away points from my perspective.

• Storing BLOBs in SQL is easy…from an application perspective and from a back-end management perspective. For small files reading and writing to/from SQL is faster than opening a file on a file system.
• Storing BLOBs in SQL doesn’t scale to the levels that customers may require. Large file streaming is faster from the file system.

The report’s objective was to establish what ‘small’ and ‘large’ mean in these scenarios.

Bear in mind that when externalizing SharePoint content we need to balance a few specific factors…

• We want the maximum size of the SQL data store to stay below 200GB if possible.
• We want the data to be in the most efficient location for the conditions.
• We want to make the SQL instance practically manageable from an IT perspective.

The conclusions from the technical report are…

• Files less than 256KB are more efficiently stored in the database
• Files larger than 1MB are more efficiently stored in the file system
• Files between 256KB and 1MB could be stored in either – if in doubt store them on the file system.[i]

How it might affect your decisions when externalizing SharePoint content from SQL.

• If you have very few large (>256KB) files then you might not want to externalize any content from SQL – just leave it alone…assuming the total size doesn’t greatly exceed 200GB.
• If the majority of your BLOBs are large then externalize everything.
• If you have a variety of file sizes then ideally you should filter data based on both the individual file sizes and the need to keep the overall SQL data store total below 200GB.

The conclusion from the report is best summarized in this one paragraph…

“The study indicates that if objects are larger than one megabyte on average, NTFS has a clear advantage over SQL Server. If the objects are less than 256 kilobytes, the database has a clear advantage. Inside this range, it depends on how write intensive the workload is, and the storage age of a typical replica in the system.”

---

[i] The ambiguity arises because in a new system files between 256KB and 1MB are better stored in SQL but over time as fragmentation occurs the performance of file management in SQL degrades faster than the file system (SQL doesn’t have any de-frag capabilities).

Check out SharePointGovernance.org – SharePoint Peer-to-peer Exchange

Take a look at SharePointGovernance.org, it is a peer-to-peer information exchange started by by Russ Edelman, founder of Corridor Consulting in association with AIIM. The site is funded through sponsorship from vendors in the SharePoint space but I can tell you from personal experience that there is a clear delineation between the content on the site and anything vendor specific.

I’ve taken one of my series of Blog posts on externalizing SharePoint content from SQL Server and turned it in to a PDF. This is the kind of thing that you might expect to see there – basic, useful, real-life SharePoint-related information.

Here are some examples of the papers that they have already collected…

• “SharePoint and ECM: Better Together” by Alan Weintraub
• “Best Practices for Forming a Governance Committee” by Karuana Gatimu
• “SharePoint versus Oracle UCM” a LinkedIN Q&A discussion around SharePoint and Oracle UCM
• “SharePoint Governance Chaos” by Russ Edelman

There are also a number of SharePoint ‘gurus’ – including yours truly (I will be referring to myself in the 3rd person as Guru Chapman from now on) who will be contributing to the site and fielding questions in the future. The list includes many of the usual suspects, John Mancini, Russ Edelman, Jeff Borghoff, Alan Weintraub, Barclay T. Blair and many other experts and bloggers

Like all peer to peer sites, SharePointGovernance will only be successful if people contribute and consume the content so take a look and let us know what you think. I’ll share any Blog comments with the other Gurus.

Faithfully yours,

Guru Chapman

Information Management Reference Model (IMRM).

I recently joined the working committee for the Information Management Reference Model (IMRM). The team is newly formed so there’s not too much to see on the Web site but I’d suggest keeping an eye on the progress that the team makes. The IMRM team is made up of customers and vendors; from what I’ve seen so far, it is focused very clearly on a nice simple agnostic model. The official goal of the IMRM Project is “to provide a common, practical, flexible framework to help organizations develop and implement effective and actionable information management programs.”

What’s the point? Don’t we already have ECM interop standards failing all over the place? Indeed a skeptic might agree but this isn’t an interop or an architectural standard, it isn’t CMIS. It is more about creating a model to provide “common language and reference”. For example, if you look at the Electronic Discovery Reference Model (EDRM), you will find that most vendors have embraced this and will talk about where their solutions play in the terms shown in this common model. If someone says “we have an issue in the Preservation area” then everyone knows what they mean and the vendor’s solutions magically become a panacea for that area of the process.

Just FYI, the official project description is… “The IMRM Project aims to offer guidance to Legal, IT, Records Management, line-of-business leaders and other business stakeholders within organizations. The Project seeks to facilitate dialogue among these stakeholders by providing a common language and reference for discussion and decision-making based on the needs of the organization.”

I’ll Blog about the progress that we make so there’s something to look forward to!

SharePoint/ECM Interop – Just like High School Sex?

Sometimes it seems to me that really getting SharePoint to efficiently interoperate with traditional ECM solutions is a bit like high school sex. Everyone is talking about it, some people claim to be doing it but in reality it isn’t happening… Maybe I am being overly cynical but I have noticed that most people that I talk to, and most articles that I read, hypothesize about how to make the systems work rather than talking about real-life successes.

Are you doing it? For real?

To be clear, I’m talking about SharePoint/ECM interop in a real production environment. If you are a customer who is doing this, especially with Documentum then do me a favor and email me. If you are a partner with a practice focused on this – or even just considering moving in to this area email me too. I’m looking to interview a few customers and partners about their experiences and/or plans just to confirm some business planning that I’m working on.

Just a thought – might a Blog title with SharePoint and Sex in the title trump all others? I’ll twitter about this article and see if I can beat Ashton Kutcher at his own game.

Moving content out of SQL Server - Conclusion

This is the conclusion to a series of articles that discuss the benefits of Moving Content out of SQL Server. Pairing SQL Server with a file system, with an archive, with a traditional ECM solution and not putting it in SharePoint in the first place.

Not surprisingly, the conclusion is that there are a number of different options when it comes to how to externalize content from SQL Server, the more  functionality you want the more complex and expensive the solutions become…no surprise there then, I guess if the cheapest solution gave you the most functionality then I’d be wasting my time writing this.

In summary, the options are…

1. Dump the unstructured content on to the local file system from SQL Server - this gets the content out of SharePoint and relieves any SQL Server bloat related issues but you do not get any additional value out of the practice.
2. Externalize all content types and store them in an archive alongside your other archived content - as well as solving the SQL bloat issues this adds significant value with common policy management, long term archiving and retention/disposition/litigation support on the archived content.
3. Store unstructured content in your ECM system alongside your existing content – fixes SQL bloat; provides policy enforcement, long term archiving and compliance management plus the ability to re-purpose and reuse the content. Also gives you a natural integration point in to other enterprise information systems
4. Leave the content in the ECM system and use SharePoint as a portal – excellent solution for accessing existing ECM content and processes but doesn’t really leverage SharePoint’s native capabilities.

An interesting question is, “How many of these will still exist in 5 years time?” I’ll stick my neck out here and suggest that only the first will no longer be around in 2014. My rationale is that #1 solves an issue caused by a questionable architectural decision made by Microsoft, if they decided to fix that then #1 would no longer be relevant. Options 2 and 3 actually add significant value by allowing you to bring SharePoint-created content in to your enterprise’s other information systems. If SharePoint is going to be the dominant solution in the space then it needs to play nicely with other solutions in your company; it becomes another ubiquitous data source/consumer. #4 will hang around simply because SharePoint is a good portal choice not because it is a good portal platform but because it is pervasive and familiar to end users.

Moving content out of SQL Server – don’t put it there in the first place.

This is the fourth in a series of articles that discuss the benefits of Moving Content out of SQL Server. The first discusses pairing SQL Server with a file system, the second with an archive, the third discusses pairing SQL Server with a traditional ECM solution; this entry looks briefly at another option – not putting it in SharePoint in the first place…

I’ll keep this super-brief primarily because I’m trying to get the articles posted before EMC World and I’m sat on the flight to Orlando right now!

Bottom line, if storing specific content in SharePoint is causing you pain then you might want to consider putting it directly in to your ECM system and then accessing it from SharePoint using Web Parts. With this approach you are using SharePoint as a portal and directly accessing the ECM’s content and processes. It is not without issues – specifically SharePoint does not know anything about the objects in the ECM system, you cannot use any of SharePoint’s native capabilities to control or interact with these objects. I’ve written plenty of other entries about this and other similar pairings but included it here for completeness.

This approach is especially effective for getting access to content and processes that already exist within your existing ECM systems. Also, it can be combined with the ECM back end concept to allow you to interact directly with content that has been externalized in to the ECM system – this is more complicated that it sounds so I’ll try to expand on it in the near future.

Rumor has it that SharePoint 2010 will support CMIS…using CMIS to connect a SharePoint front end to a ECM backend looks promising and I’m sure that I’ll be writing plenty of other articles about that one as we learn more.