SharePoint Dragons

Nikander & Margriet on SharePoint

Tag Archives: Tools

JSON Viewer

We like the JSON Viewer tool for seeing JSON data in a more user-friendly way: http://jsonviewer.codeplex.com/ . Alternatively, if you don’t want to install a tool just go to the following online JSON viewer: http://jsonviewer.stack.hu/ .

SPC 2012 Download sessions

First a disclaimer, to be able to do this, you need to have a set of login credentials in order to access the SPC 2012 sessions…

We have to say, the UI for the SPC 2012 web site and interface for downloading all sessions is, how to put this nicely? Well, it’s probably never designed for human beings. Luckily, if you’ve been to SPC2012 and want to get all the sessions, there is a solution that’s targeted towards non-human beings: http://nubo-corp.com/patrick-lamber/2012/11/spc-2012download-all-ppt-in-a-bach/ . As it turns out, PowerShell digs the UI more than we do (but we do feel sorry for the person creating the script containing all session names!).

Web PI for getting WAW

On SPC 2012, we saw a presenter use the Microsoft Web Platform Installer (Web PI) tool to get the latest version of Windows Azure Workflow (WAW), a required component if you want to run SharePoint 2013 workflows (btw, those workflows are not available for SharePoint Foundation), so we checked it out at http://www.microsoft.com/web/downloads/platform.aspx. Apparently, the tool is appalicious or apptastic or something, and makes getting the latest version of a wide range of components really easy.

If you take a closer look at the picture, you might notice that getting the latest version is a relative thing, since it’s still showing the beta versions. This will lead to problems later when running the Register-SPWorkflowService  cmdlet like this:

Register-SPWorkflowService –SPSite “http://myserver/mysitecollection” –WorkflowHostUri “http://workflow.example.com:12291” –AllowOAuthHttp

It will look for the v1.0 version of the Workflow client assembly, instead of v2.0. For now, it’s better to download WAW by searching for it and not do it via Web PI.

image

Colligo Briefcase

It looks like Colligo succeeded in bridging the gap between SharePoint and iPad (or iPhone) with their product Colligo Briefcase. Colligo Briefcase (http://www.colligobriefcase.com/) is a secure solution to work with SharePoint on an iPad/iPhone or iPod. Let’s take a look at some of the features of Colligo Briefcase:

  • You can view all your SharePoint files, including Office documents, PDFs, images and emails with the appropriate icons.
  • You can edit files, lists and metadata, even while offline and sync these changes when you’re online again.
  • You can choose what you want to sync, so you can choose to sync only the documents that you need.
  • You have a Favorites option where you can add documents that you want to access very often.
  • Using the search capabilities makes it easy to find content.

Installing and configuring Colligo Briefcase is rather simple. You can find Colligo Briefcase in the App Store. After installing the tool you will find an icon on your device. Next is the configuring part, by clicking the Colligo Briefcase icon you will be able to add a SharePoint site. The basic things you have to know are: Url, Username and Password. But that shouldn’t be a problem because you’re using SharePoint all the time ;-). You can also set the sync interval (there are several choices), the storage limit (from 20 MB to Unlimited) and the lists to sync (which lists do you want to sync on your device). After configuring these settings and clicking Done Colligo will start syncing your content:

clip_image002

Using Colligo Briefcase is very intuitive. On the left you will find the lists (as you’re used to have in SharePoint). By selecting a list you will get an overview of the contents of that list on the right. Okay, it is not SharePoint but it works almost the same. You have to get used to it but that will take only 30 minutes and after that you can work on your iPad in your SharePoint site.

A cool feature of Colligo Briefcase is the preview option. You can select a document library, click on a document and click Preview. You can also read the full version but a preview version is handy to know if you’ve got the right document.

clip_image004

If you are looking for a product to work with on your Apple device, you should definitely take a look at Colligo Briefcase. Right now Colligo Briefcase has a special launch price of $2.99, plus it’s a universal app so you can install it on both your iPad and iPhone!

Audit trail and LOGbinder SP

Nowadays, compliancy rules dictate companies to keep a track on what’s happening within their SharePoint environment. And if it’s not compliancy rules, sheer curiosity can be a sufficient driver for the need for knowledge as well. If we would get a penny for every time somebody asks a question in the SharePoint forums like: “How do I know who read document X?” (the answer being: check the audit trail), well uhm, we probably could buy us a Happy Meal or something. Whatever the reason may be, SharePoint provides the ability to introspect itself via its Audit Trail mechanism, check out http://technet.microsoft.com/en-us/library/cc824909.aspx for more details. However, based on our experience, customers find the Audit Trail useful/essential, but lacking in several ways:

  • Power users find that reports are quite unreadable. And guess what? We totally agree.
  • Power users find it a disadvantage that reports are only available in the form of Excel files. Some browser based reports or the ability to export those to XML would be nice.
  • Power users find that after time passes by, it takes a long time to generate reports. Of course this is caused by the fact that the audit trail log is growing rapidly, more on that later.
  • Administrators are not sure where the audit logs are stored. When they do find out, usually they find that storing audit trail data in the SharePoint content database doesn’t make sense.
  • Administrators are not sure if and when audit log data is purged.
  • We’ve found that the rapidly growing AuditData table (the database table that houses the audit trail data), stored in the SharePoint content database, can become a management liability. Interestingly enough, a blog post ( http://blogs.msdn.com/b/subhajitc/archive/2009/05/20/tip-the-untold-story-of-audit-logs-in-sharepoint.aspx ) mentions that an average of 10,000 hits/day can lead to a monthly increase of the SharePoint content database of 15 GB.

For an important feature, these are too many drawbacks. Are these problems solvable? Of course. Are these problems solvable in a generic way at a reasonable cost? Well, in fact, that’s the reason we’re writing this blog post. We came upon the commercial tool LOGbinder SP ( http://www.logbinder.com/ ) and it solved our Audit Trail problems. Let’s take a closer look at some of the features of the LOGbinder SP tool:

  • Creates easy to read reports.
  • Allows audit trail data to be exported to other data sources, such as the Windows Event Viewer.
  • Allows you to configure audit trail settings for multiple site collections.
  • Allows you to purge the audit log automatically.
  • Leverages existing SIEM/Log Management solutions.

Tour of the Tool

Let’s conclude the blog post with a quick tour of the tool (please refer to the product documentation for a detailed overview, if interested). Installing it and configuring is easy. LOGBinder SP consists of a UI tool and runs a separate service that collects (and optionally purges) audit trail data without affecting anything in the SharePoint installation, running in lower priority mode than SharePoint services, therefore conserving server resources when they’re needed. The only drawback we’ve found is that it’s a bit hard to change the user account under which LOGbinder SP runs.

After installing LOGbinder SP, you need to configure a thing called the Input. Here, you can specify a Default Audit Policy that can be reused and defines which events should be tracked in the audit trail. Alternatively, you can define specific audit settings for a specific site collection. This is shown in the next Figure.

clip_image002

After that, you’ll also need to configure Output. This basically decides where audit trail information should be redirected to (for example, the Windows Event Viewer) and what data will be included. This can be seen in the next Figure:

clip_image004

There’s a Service section that allows you to configure and start the service responsible for collecting (and optionally purging) audit trail data. It’s real easy to configure, but we’ve skipped a screenshot. The LOGbinder Diagnostics Events section makes it very easy to get insight into the activities of its service:

clip_image006

Finally, the most satisfying part of the tool, there’s a wide range of ootb reports available, check the following Figure:

clip_image008

For example, this report provides an overview of View events:

clip_image010

And here’s another one that concerns itself with document update events:

clip_image012

Conclusion

The audit trail mechanism in SharePoint is without a doubt an important and powerful feature of the product. Especially when doing implementations for large companies, legal firms or legal departments, we’ve found that the lack of an adequate audit trail can be a deal breaker. However, the current version of the audit trail is lacking in some ways. This can be solved by adding custom work, but that will be costly. A far better option is the LOGbinder SP tool, which currently is our favorite way of making the audit trail feature mature and complete.

The Migration Dragon for SharePoint 2010 – Migrate from the File System to SharePoint 2010

It is quite common that, during SharePoint implementations, end users want to migrate file and folder structures located on the file system to a SharePoint document library. Out of the box, you can do this by uploading multiple documents or via the Windows Explorer view. Both ways are quite easy to use but have several disadvantages:

  • When migrating larger amounts of data both methods are not really reliable. Failure usually means starting all over again.
  • Both methods don’t offer adequate insight on current progress.

Another approach would be to use third party migration tools. There are very good ones out there, but they have disadvantages too:

  • Obviously, they cost money.
  • They’re usually less easy to use (compared to the previous method).
  • They’re typically used by administrators, which leaves end users wanting to migrate files on the file system themselves in the cold.

Another approach would be to use the Microsoft SyncToy (https://sharepointdragons.com/2012/04/10/synctoy/) tool. SyncToy is a tool that synchronizes files and folders between locations using the Microsoft Sync Framework 2.0. Luckily, by mapping a network drive pointing to a SharePoint document library (http://grounding.co.za/blogs/neil/archive/2008/08/02/using-synctoy-to-synchronize-offline-sharepoint-documents-on-vista.aspx), you can use SyncToy to migrate files from the file system to SharePoint 2010. When doing this, SyncToy leverages the SharePoint Sync Framework API (http://msdn.microsoft.com/en-us/library/ee538641.aspx) to sync the file system to the SharePoint list. Or, to put it in other words, SharePoint 2010 has its own sync provider which can be consumed by clients (such as SyncToy and most notably, SharePoint Workspace 2010): http://www.chakkaradeep.com/post/SharePoint-2010-ListsGetListItemChangesWithKnowledge-Method.aspx. The SyncToy has disadvantages too:

  • It synchronizes files and folders (two-way copy), that’s something quite different from migrating files and folders (one-way copy). For larger structures, this adds considerable overhead and complexity. You should avoid synchronization, unless that’s really what you’re after.
  • It requires a network drive mapping. Most end users won’t know how to do this, and usually in larger organizations end users won’t be allowed to create them anyway. For such organizations it simply won’t be feasible to let admins create network drive mappings whenever end users need to migrate something to a SharePoint library.
  • It’s easy for end users to make a mistake. For example, renaming or deleting files in the SharePoint library may have severe consequences for the original file system after sync’ing both locations anew, without the end user realizing this.

That’s were the Migration Dragon for SharePoint 2010 comes in. Its first and current incarnation is an experimental one and it’s based on a simple idea: what would happen if a tool leverages the SharePoint managed client object model to upload files to SharePoint? The client OM has the unique ability to batch commands, so it’s very possible that the results will be remarkably good.

So, combining that with my current preferences about what a migration tool should do, Migration Dragon for SharePoint 2010 offers the following features:

  • It allows you to specify the batch size of files that are to be uploaded. For example, if you specify a batch size of 20 MB, the tool will create a batch of files up to 20 MB and then upload that batch in one go before proceeding to the next batch.
  • It allows you to specify the max allowed batch size on the server. By default, the max allowed batch size is 3 MB for the client object model which isn’t a very useful amount. If you try to upload a batch of 20 MB of files, while the server only accepts 3 MB at a time, such requests will fail miserably.
  • It’s a tool that can be used by clients. Because it uses the SharePoint client object model, the tool doesn’t need to be executed on the server (except for the part that sets the server max batch size, that functionality is not available on a client because it leverages the SharePoint server object model).
  • The tool tracks which batches failed to upload correctly. After processing every batch, it will retry to upload failed batches. In order to do so, it switches strategy. Instead of uploading the entire failed batch, which will likely fail again, it will upload each file within the failed batch individually.
  • After retrying all failures, the tool provides feedback on the files that were unable to be uploaded, even after retrying.
  • It provides accurate updates about the progression informing the end user about the number of folders that have been processed and the amount of data that has been processed. One of the ways that the tool divulges this info is in the form of progress bars, so it’s made quite clear how much work there is still left to do. This is shown in the next Figure.

    image

  • It translates most of the problematic characters/combinations in file and folder names. Problematic that is, for web environments. This includes the following tricky characters:  ~ # % & * : < > ? / { | }. I say most, because I’m not correcting some faulty combinations (such as a file name starting with dot (.), e.g. .MyFileName.txt).

Let’s take a look at the Migration Dragon for SharePoint 2010:

image

Max Batch Size

Let’s talk about the max batch size some more. The default max batch size in the tool is 10 MB, which means that all files on the file system will be packaged in a batch of up to 10 MB. That is, unless you have files that are larger than the max batch size (otherwise those files could never be uploaded). You can specify this here:

image

Since the default max batch size for the SharePoint client object model is quite small, 3 MB, it’s very likely that you want to change this. That is, if you want to gain any benefit from the batching mechanism. You need to increase this number to:

  • At least the size of the max batch size you want to use.
  • AND At least the size of the largest file you want to upload.

You can change the server max batch size by clicking the Increase button:

image

You can only execute this button when placing the tool on the server, as this part of the code uses the SharePoint server object model. Right now, if you click it on a client, the tool crashes badly. The rest of the tool can be used on any client. It executes the following code:

private void IncreaseMaxReceivedMessageSize()
{
    int increaseSize = Convert.ToInt32(txtBatchSize.Text) * 1024 * 1024;

    SPWebService contentService = SPWebService.ContentService;   
    contentService.ClientRequestServiceSettings.MaxReceivedMessageSize = increaseSize;
    contentService.Update();

}

Please Note You have to restart the web server on the SharePoint WFE(s) in order for the change to take effect.

Client requirements

The tool needs access to the SharePoint client OM. It’s included in the Migration Dragon zip file. The user account under which the tool runs needs to have appropriate permissions to upload all files and folders.

Testing

I did my testing with a corpus of 197 MB of IIS log files. We changed the batch sizes, and after repeating the tests several times, the results on our dev machine were roughly as follows:

Batch Size Processing Time
1 MB 52 secs
3 MB 50 secs
10 MB 50 secs
20 MB 42 secs
30 MB 42 secs
50 MB 60 secs + becomes error prone
100 MB 62 secs + becomes error prone
200 MB 80 secs + becomes error prone

As you can see, varying the batch sizes didn’t result in that many differences, but that’s to be expected when uploading files located on a dev machine to SharePoint on the dev machine. Things will only get interesting when using the tool on various clients.

What now?

It’s an experimental version. I have yet to start to use this tool in various environments. I’d like to get feedback on how well this tool is doing, and how much difference it makes when varying batch size. Feature requests are welcome too. You can download it here: http://gallery.technet.microsoft.com/The-Migration-Dragon-for-628acae0

SharePoint Max Dragon – checks for capacity planning limits

Enabling the workflow DSL for a custom acitivity

Here’s a small trick for enabling the workflow DSL for a custom activity:

  1. First, create a new empty SharePoint project.
  2. Add a  reference to the System.Workflow.Activities assembly. It’s located in the GAC, and therefore it may be a bit of a hassle to reference it. For cases like these, we like to use Muse.VSExtensions ( http://visualstudiogallery.msdn.microsoft.com/36a6eb45-a7b1-47c3-9e85-09f0aef6e879 ) from the Visual Studio gallery. It adds a new menu option called Add GAC Reference that allows you to search for and reference assemblies in the GAC directly. If that’s not enough, it also adds a menu option that removes unused references. It’s a great tool, that works like a charm.
  3. Also add a reference to System.Workflow.ComponentModel.dll, also located in the GAC.
  4. Add a new class: 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Workflow.Activities;

namespace SharePointProject1
{
    public partial class Class1 : SequenceActivity
    {
        public Class1()
        {
            InitializeComponent();
        }
    }
}

Finally, add another class called class1.designer.cs:
 
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SharePointProject1
{
    public partial class Class1
    {
        private void InitializeComponent()
        { }
    }
}

Now, when you click class1, you’ll find that now you can drag n drop activities. See http://social.technet.microsoft.com/Forums/en-US/sharepoint2010programming/thread/23f11f0f-d50d-4f99-a409-90f582b19442 for more info.

SharePlus: Shattering Ambitions of Aspiring Vendors

Most customers and SharePoint consultants share a vision, or dream if you will, of a SharePoint work environment that can be accessed anywhere, anytime, anyhow. Lately, we’ve actually met several people (completely separate from each other) that walked around with plans of creating some sort of IPad or smart phone support for SharePoint. It seems that Infragistics SharePlus will be shattering those ambitions. It also seems that SharePlus provides a missing piece of the “anywhere, anytime, anyhow” puzzle and may just well prove to be strategically important to Microsoft as well.

What does SharePlus do?

Finally, we’ll have a good answer when our customers are asking us about a way to access SharePoint from their iPads, iPhones, Mac desktops, BlackBerry Playbooks (BlackPads), Android phones, and soon, Windows phone support will be added. SharePlus will allow you to acess SharePoint 2007, SharePoint 2010, BPOS, and Office 365. It offers a range of authentication methods, inlcuding Windows authentication, Forms authentication, HTTPS SSL and client certificates, ISA server, and MS ForeFront Authentication. Infragistics offers three versions:

  • Lite, this version is free and is primarily intended for reading documents and list items.
  • Pro, this version (at the time of writing) costs $19,99 and offers a full-blown content and document experience AND offers offline support.
  • Enterprise, the price of this version depends on the situation and amount of users. This version is meant for in house deployment within a company, and therefore offers features such as rebranding and custom on demand features.

Yes, but what does it do?

We’re listing some of the appealing features (ignoring version differences here, contact Infragistics about that):

  • It makes reading Office and PDF documents easy.
  • It has Message File support. You can open mail files (.msg and .eml) and reply to message’s recipients, forward messages, and browse attachments.
  • It allows you to navigate sites, lists, list items, and list item metadata.
  • It allows you to search within a list.
  • It offers a drag n drop UI for sorting and grouping.
  • It allows you to manage your calendar, contacts, and tasks.
  • It has offline support for lists.
  • It adds an additional level of security for your client device (such as encryption for data that is locally stored).
  • For iPad2 and iPhone: support for capturing and uploading audio and video recordings.
  • WiFi sharing.
  • Search bonuses such as searching within a user profile or within a list.

Where can I find more info about it?

For sure, SharePlus is an interesting product that has caught our attention. If you feel the same way, the home of SharePlus is http://www.shareplusapp.com/.

Maxer and Beads

We were looking for an apt metaphor to describe the need for the Maxer for SharePoint 2010 tool. At first, we were inclined to use the Big Bang theory as a metaphor to describe the common problem within a SharePoint farm in that it is ever expanding. However, the universe is a scary thing and thinking about it is too. So, we switched to another metaphor: a jar of beads. Beads are a lot nicer, they’re not scary, beautiful to look at, and you can create wondrous things with them. Don’t believe us? Check out these:

Tibetan Beads - Button Beads / Glimmering Golden Amethyst Beads - Cloudy Appearance / Lavender Purple Wood Beads - Classic Abacus / Tasteful Coffee Colours Nut Beads - Natural Coconut Shell
Bracelet Watch - Light Blue / Rhinestones Beaded Bracelets - Faceted Green / Pink Pearls Evil Eye Beads - Flecked Yellow Pieces / Eerie Red

Please note: All pictures were taken from either Google search (search term “beads”), or, with permission, from beads.

Now, imagine your SharePoint farm is like a jar of beads. You can create beautiful things using these beads, but you can also try to stuff too much beads into the jar, in which case bad things will happen. There are SharePoint capacity planning guidelines you can follow to prevent the jar from breaking, but just like a jar of beads can be hard to count, it can be difficult to determine if, how, and where you are crossing capacity boundaries within a SharePoint environment. That’s where Maxer for SharePoint 2010 enters the stage…

Maxer for SharePoint 2010 is a command line tool that checks for capacity planning limits as described per the Planning section of the TechNet Wiki SharePoint 2010 Best Practices overview page at http://social.technet.microsoft.com/wiki/contents/articles/8666.sharepoint-2010-best-practices-en-us.aspx.

The Maxer zip file can be extracted on a server of the SharePoint farm. To execute it, all you need to do is type maxer at the command line. It will then start analyzing your entire SharePoint farm (so make sure the user account executing the tool is extremely powerful, otherwise the analysis will fail) and create a text file called overviews.txt in the same directory as Maxer. Depending on the size of your farm, this may (or may not) take a long time and can be a CPU intensive operation, so be careful to plan this off hours when doing this in a production environment.

Since this is the first version, a limited set of 10 features is included. Also, error handling is quite rudimentary. Community feedback will decide the future direction of Maxer for SharePoint 2010.

The v1.0 feature set contains the following capacity limit checks:

  • A single web application can only have 250,000 site collections.
  • A single web application can only have 300 content databases.
  • A single content database can only have 2000 site collections.
  • A single site collection can only have 2 million users.
  • A single user can only be a member of 5000 groups.
  • A single site collection can only have 10,000 SharePoint groups.
  • A single SharePoint security group can only have 5000 AD principles/users.
  • A single site collection can only have 250,000 sites and subsites.
  • A single web site can only have 2000 sub web sites.
  • A single list can only have 30,000,000 items.

The top feature request that will be implemented next is to have the option to specify whether a full blown report should be generated or only a report of capacity violations. Currently, Maxer only supports the full blown report.

Maxer for SharePoint 2010 can be downloaded at http://gallery.technet.microsoft.com/Maxer-for-SharePoint-2010-8cd0f26f.