How to issue Http Web Requests to a SharePoint 2010 site with FBA and SSL

The .NET Framework provides some handy classes that help you manage HTTP communication through requests and response objects.

Everything is plain and simple, as long as you are connecting to a resource endpoint that does not require authentication and is available through an unsecure channel (i.e. HTTP). Which, of course, is not always the case.

I had a requirement to “invoke” a SharePoint 2010 resource (a file, for example) that is protected by Forms Based Authentication and is exposed by a secured SSL channel. The task is slightly more complicated, since it has to be performed by a client that has no easy way to invoke web services using an autogenerated proxy. That is, no “Add Web Reference” available. Indeed, no RAD environment at all.

You can find my solution in the code snippet below. I have written it with PowerShell just as a prototyping tool, it will have to be translated. But anyway some interesting points can be highlighted:

  • I used the .NET Fx API to “ignore” SSL certificate warnings. This may not be an option sometimes, but the solution can be extended adding some certificate chain verification or whatever you need to check. This is done setting the ServerCertificateValidationCallback static property (line 1).
  • I had to invoke the Authentication.asmx SharePoint Web Service in order to authenticate and, then, reuse the authentication cookie in subsequent requests. This is a requirement for FBA access. I handcrafted the SOAP message since there’s no easy-to-use helper/proxy (line 2-24).
  • I finally issued my HTTP request, using the authentication cookie and adding an HTTP header (line 26-28).
   1: [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

   2: $cookieContainer = new-object System.Net.CookieContainer

   3: $authEnvelope = '<?xml version="1.0" encoding="utf-8"?>

   4:                 <soap:Envelope xmlns:xsi="" xmlns:xsd="" xmlns:soap="">

   5:                   <soap:Body>

   6:                     <Login xmlns="">

   7:                       <username>yourusername</username>

   8:                       <password>yourpassword</password>

   9:                     </Login>

  10:                   </soap:Body>

  11:                 </soap:Envelope>';

  12: $encoder = [System.Text.Encoding]::UTF8

  13: $authEnvelopeBytes = $encoder.GetBytes($authEnvelope)


  15: $authRequest = [System.Net.HttpWebRequest]([System.Net.HttpWebRequest]::Create("https://yoursite/_vti_bin/authentication.asmx"))

  16: $authRequest.CookieContainer = $cookieContainer

  17: $authRequest.Headers.Add("SOAPAction", "");

  18: $authRequest.ContentType = "text/xml; charset=utf-8";

  19: $authRequest.ContentLength = $authEnvelopeBytes.Length

  20: $authRequest.Method = "POST";

  21: $authRequestStream = $authRequest.GetRequestStream()

  22: $authRequestStream.Write($authEnvelopeBytes, 0, $authEnvelope.Length)

  23: $authRequestStream.Flush()

  24: $authRequest.GetResponse()


  26: $request = [System.Net.HttpWebRequest]([System.Net.HttpWebRequest]::Create("https://yoursite/yourcontent"))

  27: $request.CookieContainer = $authRequest.CookieContainer

  29: $request.GetResponse()


New Slovenian SharePoint User Group

I wish to say “good luck” to the brand new Slovenian SharePoint User Group.

A first meeting will take place today at 5PM: Robi Vončina and Uroš Žunič will present a couple of sessions about “SharePoint Gotchas” and the “Client Object Model” respectively, and maybe some surprise is on the way 🙂

Well done guys, and hopefully we could manage something together with the Italian SharePoint Community (considering the distance… it takes me less to go from Bologna to Ljubljana than to Genoa :-))

More on invalid file and folder names

I’ve been talking about file name limitations for SharePoint 2010 resources a couple of months ago, focusing on illegal characters and path/url length.

There is, indeed, something more which is related to invalid “patterns”.

Just to give you an example, you cannot use folder names that end with:

  • .files
  • _files
  • -Dateien
  • _fichiers
  • _bestanden
  • _file
  • _archivos
  • -filer
  • _tiedostot
  • _pliki
  • _soubory
  • _elemei
  • _ficheiros
  • _arquivos
  • _dosyalar
  • _datoteke
  • _fitxers
  • _failid
  • _fails
  • _bylos
  • _fajlovi
  • _fitxategiak

Take a look at this post by Jeremy Jameson for further information.

The reason for this behavior is the special handling of the so-called “thicket folders”.

What is a thicket folder?

I’m copying here some glossary terms from MS-GLOS:

thicket: A complex HTML document may be stored as a thicket, which consists of the thicket main file along with a hidden thicket folder containing a thicket manifest and a set of thicket supporting files that together store the referenced content of the document (for example, a Web page or Office Word HTML document that includes graphics, pictures, or other media is a thicket).

thicket folder: A hidden folder containing a thicket manifest and a set of thicket supporting files that together store the referenced content of a complex HTML document (for example, a Web page or Microsoft Word HTML document that includes graphics, pictures, or other media). A thicket folder has a name based on the name of the thicket main file, with a suffix added that is recognized as a thicket folder.

thicket main file: The main file of an HTML document that references contained elements such as graphics, pictures, or other media stored as thicket supporting files in a thicket folder.

thicket manifest: An XML file containing a manifest for the set of thicket supporting files that together store the referenced content of a thicket main file, which by convention is named filelist.xml and is contained in a hidden thicket folder.

thicket supporting file: A file containing a graphic element, a picture, or other media that is referenced by a thicket main file and stored in a thicket folder as part of a thicket.

What is somehow surprising is that SharePoint does not prevent you from creating such a folder. I mean, no error whatsoever. It just renames the offending folder with a trailing underscore!

Why? Maybe just to make OOTB migration easier (automatic rename instead of manual renames beforehand).

Anyway… tricky, isn’t it?

Destinazione Office 365

Destinazione Office 365 (which, as you may argue, is the Italian name for Destination Office 365 :-)) is a new technical event organized by Green Team, together with Microsoft, AvePoint and Nintex. The event, which is free BTW, will take place in Microsoft auditoriums in Milan and Rome.

We have defined an initial schedule with two dates.

  • Milan, February 8, 2012
  • Rome, February 14, 2012

Igor Macori, Riccardo Celesti and myself will be talking about Office 365 from the different perspective of End Users, It Professionals, Site Builder and Developers.

You can find detailed information on the event web site, starting from the agenda:

  • 09:30: Welcome (Microsoft)
  • 09:45: Office 365 Overview (Igor)
  • 10:00: Administering Office 365 (Igor)
  • 11:00: Migrating to Office 365 with AvePoint DocAve
  • 11:30: Customizing SharePoint Online (me and Riccardo)
  • 12:30: Automate Office 365 workflows with Nintex Live

And… there’s something more, which I’ll be talking about in a future post.

Stay tuned, and see you there!!

SharePoint Management Shell vs standard PowerShell console

You know that most of the administrative tasks you perform on a SharePoint Farm can be accomplished using PowerShell scripting. You get a bunch of commandlets (more than 600 indeed!) that are registered through the Microsoft.SharePoint.PowerShell snapin, and you can even add your own commandlets using standard SharePoint deployment techiques.

That’s why sometime you just end up firing the default PowerShell console and typing

Add-PSSnapIn Microsoft.SharePoint.PowerShell

Well… you get something that is “similar” to the SharePoint Management Shell Smile

The SharePoint Management Shell is nothing more than a standard PowerShell console (powershell.exe) where tha snapin is automatically loaded upon startup, leveraging a startup script:

C:\Windows\System32\WindowsPowerShell\v1.0\PowerShell.exe  -NoExit  ” & ‘ C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\POWERSHELL\Registration\\sharepoint.ps1 ‘ “

The sharepoint.ps1 script is just a few lines of code:

$ver = $host | select version
if ($ver.Version.Major -gt 1)  {$Host.Runspace.ThreadOptions = “ReuseThread”}
Add-PsSnapin Microsoft.SharePoint.PowerShell
Set-location $home

In the above snippet you can notice the automatic loading of the SharePoint snapin and a “change directory” instruction that brings you to your default home directory (tipically, c:\users\yourusername).

But there’s an additional line:

if ($ver.Version.Major -gt 1) {$Host.Runspace.ThreadOptions = “ReuseThread”}

This line sets the PSThreadOptions of the PowerShell runspace to ReuseThread, which means that the same thread is used for all command line invocations since the shell has started.

The other options are (see

  • Default
    On the local computer,
    UseNewThread is used for runspaces and ReuseThread is used for runspace pools. Server settings are used for remote runspaces and runspace pools. This field is introduced in Windows PowerShell 2.0.
  • ReuseThread
    Creates a new thread for the first invocation and then re-uses that thread in subsequent invocations. This field is introduced in Windows PowerShell 2.0.
  • UseCurrentThread
    Execution occurs on the thread that called the Invoke method. This option is not valid for asynchronous calls. This field is introduced in Windows PowerShell 2.0.
  • UseNewThread
    Creates a new thread for each invocation. This field is introduced in Windows PowerShell 2.0.

If you want a proof of this behavior, just run both a standard powershell console and a SharePoint Management Shell, then run the followind command a few times in each shell.


This instruction prints the internal identifier of the managed thread (i.e. the CLR Thread object)

You’ll notice that you get the same value in the SharePoint Management Shell, whereas you get different values in the standard one. By the way, PowerShell ISE uses the ReuseThread option too.

What’s the issue with the default value, i.e. having a new thread spawn (or picked up from a pool) for every command invocation?

Well… first of all there are some SharePoint objects that are not completely thread safe, sou you should pay careful attention when using them from different threads (even if, in this particular case, you have multiple threads that are not executed in parallel Smile)

Another noticeable issue you may experience, though, is resource leakage: if you create SharePoint objects in one thread and suddenly have no reference to that thread any longer, you end up wasting resources that are not disposed correctly.

Wrapping it up… the SharePoint Management Shell is nothing special, but it’s a little bit more than adding a snap in!!

JPGs not showing in browser window

I have to preface that I’m *not* a web designer (should I send you some snapshots of the web pages I have produced, you would definitely agree with me… that’s why I’m not adding pictures to this post :-)).

Therefore the topic of this post may seem obvious to the web designers amongst you.

Anyway, here’s what I have noticed some days ago, in the context of a SharePoint 2010 intranet project where some CQWPs display the most recent items (news, etc..), along with a thumbnail image beside the item body.

Well, the final effect was that some of the images were not displayed at all in the rollup page.

After double-checking the publishing status of each item, I could confirm that each picture item was checked-in and approved, so that it should be visible to all authenticated users.

But while navigating to the assets library that contains the pictures as a site owner, the exact same behavior was visible there: even the XSLT list view web part shows the items, with pictures dispayed as a red cross.

Then, Fiddler comes to the rescue. I fired it up, trying to figure out what was the real network traffic behind the picture download. I got an HTTP 200, and the response mime type was correctly set to image/jpeg.

Ok, then there’s some issue with the image itself. I saved the stream locally, then tried to open the image with any kind of picture viewer tools and… the image displays correctly, no matter of the tool I have used.

This is a confirmation that the image type is not well supported by the browser.

It’s a browser issue, so the next step is to navigate to the image url using different browsers and try to spot the difference… and the difference was now quite evident.

IE8 had issues with that specific image, while neither FF nor Chrome nor IE9 suffered any problem.

Then, I just had to google for “IE8 JPG” to find what I was looking for.

I knew that JPG is too vague, since you can get different encodings and internal representations, ending up with a JPG file extension as well.

But I didn’t know that IE8 dropped support for jpeg images saved as CMYK.

Here you can find some information about this issue, and even a web page where you can test how your browser behaves with CMYK JPEGs:

Browser are sometimes always strange 🙂

Ahhhh, the Ribbon!

Let me say first that I definitely like the Ribbon, especially from a user interaction perspective. You have to get used to it, but we have to admit that this kind of user experience was not introduced with SharePoint 2010!

We got Office 2007 some years ago. We are seeing Windows tools that have been rebranded this way (Paint? Wordpad?). Components vendors are providing a ribbon framework since 3 or 4 years, which means that a crew of developers are releasing Windows Applications with a ribbon on top (even when it is not suggested by official or common-sense guidelines).

That said – and here my complaints start – the ribbon is quite an obtrusive component. And this is especially true when it is used in web applications, where you have a much more fragile control over the final layout your users will be experiencing. I’m a fan of the ribbon, but I still have to struggle a little bit to convince myself that it’s good as well for html rendering.

Let me expand these considerations with a couple of more practical examples.

Imagine you are realizing a public, internet-facing web site. And you do this with SharePoint, right? J

It will take you a couple of hours to realize that the ribbon should be made available to content editors, but you need it to be hidden to anonymous users (or more generally to site viewers).

A ton of articles and tutorials have been written, and it’s not so complicated as it may seem: you can use some trimming controls (SPSecurityTrimmedControl, LoginView, EditModePanel, AuthoringContainer, something you have developed yourself, etc) and just prevent it from rendering.

Will the ribbon be visible? No. Will it take part to the page markup? It depends.

The reason why it depends is strictly related to the way you are hiding the SPRibbon control.

Are you using some CSS attributes (display:none) applied to some container? Ok, go back immediately and try another approach: the markup fragment is there, and it is useless.

Are you using some server-side visibility logic (as suggested above)? Ok, the ribbon markup is not there any longer. Much better! Well done! But…

Have ever noticed that you are emitting tons of javascript code and did you ever wonder where that script code comes from?

Most of the times, ASP.NET controls generate JS code by with the help of some ClientScript.Register<Something> method. If you want to spend some time in front of a decompiler and start investigating how SharePoint controls behave, you’ll find out that many of them contain a bunch of code like the one I’m showing right below:

If (SPRibbon.GetCurrent(Page) != null)


// Write 10 KBs of ugly JS


Is this harmful? Hopefully not.

Does this execute if you have hidden the ribbon control? Probably yes.

Indeed, many conditional panels (SPSecurityTrimmedControl, for example) apply their conditional logic very late in the page rendering lifecycle (usually during the PreRender phase). And they are just preventing the ribbon to render, they are not eliminating it from the page control tree!

Which means that any control on the page that checks for the existence of the ribbon will succeed in

this check and perform whatever operation it is supposed to do.

Summing it up:

· The ribbon is hidden

· It does not occupy bytes inside the page body by itself

· It has “side effects”, causing other controls to emit extra markup or, more generally, to behave inconsistently

How can you fix this?

You could use some smarter conditional logics (a SmartPanel, if you like the name) that could just “ignore” its contents if the condition is not met. By “ignore” I mean that this panel should not add its child controls to the control tree at all.

While I would suggest you to carefully consider this approach, I would also like to point-out some drawbacks you may face.

If you remember, I had mentioned two side effects at the beginning of this article. So here’s the other one.

The SPRibbon control automatically sets the MaintainScrollPositionOnPostback property on the page to true. Which means that, after a postback, the user navigation is taken back to the same vertical scroll position he had set before.

Is this good or bad?

Well, it depends, of course.

But you have to take care of this behavior, since the sole presence of the Ribbon control may vary this after-postback setting: if you hide or show the ribbon based on the user rights, you end up having users with different rights having a different navigation experience. And… this does not depend, this is bad by definition J

SharePoint Internet Sites – Performance Optimization for Data Access (1)

Then it comes to displaying something on your pages!

I presume that you are building a dynamic web site 🙂

By “dynamic” I mean that its contents comes from a data storage (SharePoint being one of these) where it is written in a serialized format used for persistence. These data are then extracted from the data store during the request processing phase, and some sort of transformation is applied to make them presentable as HTML markup on your final pages.

This process (extract => transform => render) can be extremely slow, and you may guess the reason for this consideration: if you wish to load a million rows from a database table (or 4K rows from a SharePoint list), then transform the resultset in XML format, then apply a complex XSLT transformation that finally produces 4 bytes of HTML markup, it’s clear that something is missing from your architecture design!

But even if you optimize every single step in the above process, you may end-up with a CPU and memory consumption that is excessive in heavy load scenarios.

The answer seems obvious: the best way to reduce the time required to load data from a persistent storage is… just not reading anything at all!

That is, use caching!


Cool, you may say, and while saying “cool” you enable output caching on every page of your portal.

It will take just a couple of minutes for you to receive a phone call be your customer, saying that the users are complaining about strange behaviors during navigation. For example:

  • “I logged in but the name that is displayed on the top of the page is not my name”
  • “I did not add any item to the cart, but suddenly I see the cart is filling up with articles I’m not even interested about!”

Well, caching has its own drawbacks, for sure.

Here’s a list of pain points you need to be aware of:

  • Cache data should be saved somewhere, and it will consume resources
  • Windows processes do not share memory (unless you do this explicitly, which I don’t suggest anyway), so in a multi-server scenario you get duplicated information (one copy of each data set for each process serving http requests)
  • Sometimes you end up having multiple processes, even on a single WFE server topology (this is called web gardening)
  • If you choose to externalize data to a common, shared location, you probably need to consider data serialization as a limitation (you can save a string, but you cannot save an XslCompiledTransform instance, just to give you an example)
  • Once you put data into a caching location, this data becomes old, unless you implement a valid cache invalidation mechanism
  • This cache invalidation mechanism is often hard to implement
  • Coding can be tricky
  • Coding can be error prone (you should never rely on a copy of your data being available in the caching storage)

This list is by no mean a suggestion to avoid caching. On the contrary, I strongly suggest you to apply caching whenever it fits.

Therefore, I would like to summarize what SharePoint offers OOTB, trying to provide you some best practices in each case.


You get three different flavors of cache in SharePoint 2010.

Here’s a small diagram that display them, giving you some background that we will use later to discuss about when you should use any of these techniques.

Object caching

In a word: use it!

SharePoint uses it by default as an optimization for some key components of a typical web site (ContentByQueryWebParts, Navigation structure, etc…).

You should just be aware that some query filters (for example, one based on the current user) makes it not applicable (and indeed the site query engine prevents caching in these situations).



I would encourage you to use object caching when you write code against the SharePoint server object model.

How? You cannot explicitly query the cache structure, but you should use classes (SPSiteDataQuery, CrossSiteQueryInfo and CrossSiteQueryCache) that can do the hard work for you. This is transparent, which is fine since you can forget about check for null data or stale data: everything is under the control of the Cache Manager.

Output Caching

In a word: always consider output caching while designing and developing pages and page components, and try to apply a design that makes output caching applicable.

A little example could be helpful in this case.

Imagine you have implemented a page layout that displays a lot of aggregated data coming from external resources. This data takes quite a long time to load, and the presentation layer takes some time to render it too. Plus, this data does not change very often, so you should not worry about invalidation.

This is a perfect candidate for output caching, unless for a very small portion of the page layout, more specifically a box that displays weather information reading it from an external RSS service, filtered by the location that a user has specified in his profile settings.

If you apply output caching to the page layout, every user will see the weather for a single location (the one of the first user hitting the page), and the weather will be constant for the whole duration of the page layout caching time.

This should not be an obstacle to applying output caching to the page layout. How can you do this?

Here’s a coupe of possible approaches:

  • Use a combination of AJAX requests and JS elaboration to read information “on the fly” and transform the page accordingly. The html code of the page can be “weather ignorant”, since the only pieces remaining there are an empty container and the client script code that issues the asynchronous HTTP request and parses the results producing the final markup. And both the empty container and the script code can be cached!
  • Use Post Cache Substitution. This is a somewhat complex technique (I mean, it’s easy for simple tasks, but it may get tricky easily). In a nutshell, you register a control for post cache substitution, and the ASP.NET runtime calls back your control asking for a string value the it will insert into the page exactly where the control markup had been rendered, replacing it with something else. The page keeps being cached, although part of it are indeed recalculated for every request.

Blob Caching

I’m mentioning Blob Caching here for the sake of completeness. But I would like to point out that it is not at all related to data or markup caching, so it does not reduce the computation and rendering time of a page “per-se”. It creates copies of static resources (css, js, images, etc… you can specify the resource by extension) that are saved to the file system of each web frontend server. An http module is responsible of the resource retrieval, effectively bypassing need for the document to be loaded from SharePoint (then from SQL, which is expensive if compared to raw filesystem access).

I’m going to talk about Blob Caching in a future part of this articles series, but I hope that this was enough to explain at least what blob caching is, especially compared to the other available caching techniques.


That said, what tools can help you investigate data access issues related to caching?

Here I’ll name a few, but consider that this list is by no mean exhaustive.

  • SharePoint logging
    • ULS logs contain information about Cross Site Queries, which may or may not use caching
    • Logging database for blocking queries reports (a blocking query is a good candidate for substitution with some data access logics)
  • Developer Dashboard
    • You get the execution time at a very detailed level, which may help you investigate which part of the page lifecycle needs further optimization
    • If you are a developer, you can use the SPMonitoredScope for instrumentation
  • Performance counters
    • Monitoring resource consumption you may discover that you need some caching optimization
    • ASP.NET provides several counters related to its Cache Engine
  • DbgView
    • You can output trace messages that will be consumable even on a live production server. This is not related to caching by itself, but it can definitely be a useful companion

Efficiently purge large lists

Sometimes you may need to clear the contents of a huge SharePoint list.

You can do this through the web user interface, but you will probably face some limitations: bulk operations are limited to 100 elements, the folder structure may be an obstacle, etc…

If you prefer writing script code (and if you have remote access to a farm server), you may gain quite a lot of time.

Here’s how:

function ClearList([Microsoft.SharePoint.SPWeb]$web, [Microsoft.SharePoint.SPList]$list)


    $sbDelete = new-object System.Text.StringBuilder

    $sbDelete.Append("<?xml version=`"1.0`" encoding=`"UTF-8`"?><Batch>")

    $command = "<Method><SetList>" + $list.ID + "</SetList><SetVar Name=`"ID`">{0}</SetVar><SetVar Name=`"Cmd`">Delete</SetVar></Method>";

    $items = $l.Items

    foreach ($item in $items)


        [void]$sbDelete.Append([System.String]::Format($command, $item.ID.ToString()))







    catch [Exception] {

        Write-Host $_.Exception.ToString()



This can be used just for SharePoint list, for document libraries you should use a slightly different approach, anyway…

Hope useful!

File name limitations

Based on my experience with customers, training attendees and, more generally, end users, if I ask “What is SharePoint” the answer I get back most of the times is “It’s a document management system”.

Of course it is – also – a document management system. I presume that DM features, being there since the very first version of the product, connotate the product especially for people who do not know it in depth.

That said, and even if you want to consider it just a document repository, it requires you to obey a set of rules for files and, more generally, folder names.

Most of these rules are a consequence of the fact that a file/folder name *is* the file/folder url, and as such you cannot have anythiing in there Smile

Here’s a short list of the limitations you should consider when defining names

Forbidden characters.

You cannot use any of the following characters:

  • /
  • \
  • :
  • *
  • ?
  • <
  • >
  • |
  • #
  • <TAB>
  • {
  • }
  • %
  • ~
  • &

Plus, the last character of the file name should never be a dot (.).

File name length.

The file name (just the name, not the full path) should be less than 128 characters, including the file extension.

Path length.

The full url of the file/folder should be less than 260 characters.

And maybe

Beyond these basic rules, there may be something else that is not strictly required, but could immensely help avoid bugs: for example, you can include a + character in a file name, but you will break any code that does not escape urls correctly (a plus character in urls is translated to <SPACE>). You may argue that this is not your fault, if something breaks because it doesn’t handle escaping correctly. And you are right… but anyway, sometimes it’s just a matter of convenience Smile