Saturday, May 24, 2014

Managing NuGet packages dependencies with the Package Visualizer tool

If you ever used NuGet on a large enough solution you know you can get into trouble when projects reference different versions of the same NuGet package. That happens a lot in Azure projects as the libraries/packages get updated all the time.

I'm really surprise when I talk to people using NuGet everyday that they don't know about the Package Visualizer tool. (update: I've been told that this feature requires VS Ultimate and is not available in the Pro version. I'm still going to show it to you but stay tune for another post with a free alternative later)

NuGet Package Visualizer in Visual Studio


Once you open up a solution in Visual Studio you can go to the Tools menu, NuGet Package Manager and Package Visualizer.


This will analyse all the packages.config files in the solution and generate a DGML diagram of all NuGet packages and projects of the solution.  The diagram will help us see packages usage in the solution and find the ones with different versions.  Below you can see that I've tried this on the Roslyn (open source C# compiler) solution.


The first thing to note (and a surprise to me!) is that Roslyn use the XUnit testing framework and not MsUnit! More seriously we can quickly see that we have no duplicate packages with different versions. If we compare that to this sample solution I created we can see I'm using two versions of the Json.NET library. Now I know I should update the ClassLibrary1 project to use the new version of the package.


Of course, this only work for NuGet packages but it would be useful to have something like this for regular DLL references.  I'll try to work on a LinqPad query to generate such a DGML graph with all projects, libraries and packages.  Stay tune till next time.

Friday, April 25, 2014

Looking inside a NuGet package with NuGet Package Explorer

When I really want to learn something new (like a new tool, technology or a programming language) I do two things

  • I try it myself
  • Check out what others have done

I'm currently learning how to create my own NuGet packages so I'm trying to do a lot of things on my own, but I would also like to see how existing packages are made.

After installing a NuGet package in your project you can go in the packages folder and unzip the .nupkg file (yes, it's only a zip file with a different extension).  Fortunately, there is an easier. NuGet Package Explorer is an open source tool available on CodePlex.

With it you can load a package from the official NuGet feed or any other feeds you want even local feeds.


Then when we open a package we can explore its content and even go inside individual files



NuGet Package Explorer also allow us to edit files and the package itself if we want.

One trick I like to do is to add my local NuGet package download cache as a feed. To do that in the Tools menu select View NuGet download cache.



This is the folder where all the packages you previously downloaded are cached (from Visual Studio, NuGet Package Explorer and any other NuGet based tools). Simply copy the path and paste it in the Package Source field like this



This way I can quickly get to a package I just installed in my solution.

NuGet Package Explorer is a powerful tool I use a lot to understand how NuGet packages are made.

I hope it will help you too.

Friday, April 18, 2014

Packaging and distributing tools using NuGet

NuGet is an amazing tool to manage dependencies for external libraries. As part of Visual Studio it gives us an easy way to install and update those libraries over time. Now even Microsoft use it extensively to release updates to us. But the power of NuGet doesn't stop here. Some people found imaginative ways to use NuGet like the ScriptCs project and Chocolatey.

Another usage is to package and distribute utilities via NuGet packages like FAKE and xUnit.Runners. To create your own tool package you need to author a NuSpec file like this:

<package>
  …

  <files>
    <file src="tooling\app.exe" target="tools\" />
  </files>
</package>

The key here is to set all your file's target to tools\. Doing that the package will be considered a solution-level NuGet package and will be available solution wide instead of only for one project.

For example if I install the xUnit.Runners package to my solution like this:

PM> Install-Package xunit.runners

You will see that only a new package.config file will be created in the .nuget folder at the root of your solution. This is where all solution-level packages will be referenced. Nothing will actually change inside your projects.

After that all the files required to run the xUnit runner from the command line, PowerShell or a build script will be available from the \packages\xunit.runners.1.9.2\tools folder.

The power of NuGet doesn't stop there but we'll check that in another blog post.

Sunday, December 29, 2013

Implementing a Treemap in C#

I was wondering how to implement a treemap in C#. A treemap is a data visualisation technique that looks like this



I know a few examples exist online but between a very abstract paper like this one and a javascript implementation on GitHub I was wondering does one could write an algorithm from scratch using a naive approach.

Let's start with a simple example, the geometric series 1/2 + 1/4 + 1/8 + 1/16 + 1/32 + 1/64… would give us something like this



My guess would be to use a recursive algorithm to create our treemap. First we need to order all the elements from biggest to smallest. Then we start to divide the area for each elements starting with the first element which takes half the available space. After that we divide the remaining area with the rest of the elements. We repeat this process until we have all the pieces.

Now if we take another more realistic example



Here every time we need to slice the area we need to make sure it won't be too small or won't look so great. If the element represent 50% or more of the total it won't be a problem but what about only 10% or 4%? I think we should set a minimum threshold for our slice, let say 25% for now. We will experiment with this value once we finish our algorithm.

So what happen if the largest value is below our 25% threshold? I think we should include more elements in the slice until we reach at least 25%. For example if we have 14%, 8% and 5% the total is 27%, so our first slice will be 27% of the available space. Then we need to distribute the 3 elements in that slice. This is in essence a subset of our original problem so we can repeat the process just for that slice.

By the way what orientation the first slice should be? I think we should always slice on the longest side of the area rectangle. If we have a square then it doesn't matter really.

Next, how do we divide the first slice? We have 3 elements: 14%, 8% and 5%. If we look for a solution from the start we now have to following proportions: 51.8%, 29.6% and 18.5%. We can take a new slice only for the first element. And we repeat the process for the last 2 elements. Which is now 61.5% and 38.5%. At each step we need to evaluate if the slice will be horizontal or vertical depending on the the shape of the area we have.

Finally, all we have to do is repeat this until all the elements are placed!

Here is my implementation in LinqPad in 3 parts

Slice calculation
public Slice<T> GetSlice<T>(IEnumerable<Element<T>> elements, double totalSize, 
 double sliceWidth)
{
 if (!elements.Any()) return null;
 if (elements.Count() == 1) return new Slice<T> 
  { Elements = elements, Size = totalSize };
 
 var sliceResult = GetElementsForSlice(elements, sliceWidth);
 
 return new Slice<T>
 {
  Elements = elements,
  Size = totalSize,
  SubSlices = new[]
  { 
   GetSlice(sliceResult.Elements, sliceResult.ElementsSize, sliceWidth),
   GetSlice(sliceResult.RemainingElements, 1 - sliceResult.ElementsSize, 
    sliceWidth)
  }
 };
}

private SliceResult<T> GetElementsForSlice<T>(IEnumerable<Element<T>> elements,
 double sliceWidth)
{
 var elementsInSlice = new List<Element<T>>();
 var remainingElements = new List<Element<T>>();
 double current = 0;
 double total = elements.Sum(x => x.Value);
 
 foreach (var element in elements)
 {
  if (current > sliceWidth)
   remainingElements.Add(element);
  else
  {
   elementsInSlice.Add(element);
   current += element.Value / total;
  }
 }
 
 return new SliceResult<T> 
 { 
  Elements = elementsInSlice, 
  ElementsSize = current,
  RemainingElements = remainingElements
 };
}

public class SliceResult<T>
{
 public IEnumerable<Element<T>> Elements { get; set; }
 public double ElementsSize { get; set; }
 public IEnumerable<Element<T>> RemainingElements { get; set; }
}

public class Slice<T>
{
 public double Size { get; set; }
 public IEnumerable<Element<T>> Elements { get; set; }
 public IEnumerable<Slice<T>> SubSlices { get; set; }
}

public class Element<T>
{
 public T Object { get; set; }
 public double Value { get; set; }
}

Generating rectangles using leaf slice (slice with only one element in it)
public IEnumerable<SliceRectangle<T>> GetRectangles<T>(Slice<T> slice, int width, 
 int height)
{
 var area = new SliceRectangle<T>
  { Slice = slice, Width = width, Height = height };
 
 foreach (var rect in GetRectangles(area))
 {
  // Make sure no rectangle go outside the original area
  if (rect.X + rect.Width > area.Width) rect.Width = area.Width - rect.X;
  if (rect.Y + rect.Height > area.Height) rect.Height = area.Height - rect.Y;
  
  yield return rect;
 }
}

private IEnumerable<SliceRectangle<T>> GetRectangles<T>(
 SliceRectangle<T> sliceRectangle)
{
 var isHorizontalSplit = sliceRectangle.Width >= sliceRectangle.Height;
 var currentPos = 0;
 foreach (var subSlice in sliceRectangle.Slice.SubSlices)
 {
  var subRect = new SliceRectangle<T> { Slice = subSlice };
  int rectSize;
  
  if (isHorizontalSplit)
  {
   rectSize = (int)Math.Round(sliceRectangle.Width * subSlice.Size);
   subRect.X = sliceRectangle.X + currentPos;
   subRect.Y = sliceRectangle.Y;
   subRect.Width = rectSize;
   subRect.Height = sliceRectangle.Height;
  }
  else
  {
   rectSize = (int)Math.Round(sliceRectangle.Height * subSlice.Size);
   subRect.X = sliceRectangle.X;
   subRect.Y = sliceRectangle.Y + currentPos;
   subRect.Width = sliceRectangle.Width;
   subRect.Height = rectSize;
  }
  
  currentPos += rectSize;
  
  if (subSlice.Elements.Count() > 1)
  {
   foreach (var sr in GetRectangles(subRect))
    yield return sr;
  }
  else if (subSlice.Elements.Count() == 1)
   yield return subRect;
 }
}

public class SliceRectangle<T>
{
 public Slice<T> Slice { get; set; }
 public int X { get; set; }
 public int Y { get; set; }
 public int Width { get; set; }
 public int Height { get; set; }
}

Drawing the rectangles in WinForm
public void DrawTreemap<T>(IEnumerable<SliceRectangle<T>> rectangles, int width, 
 int height)
{
 var font = new Font("Arial", 8 );

 var bmp = new Bitmap(width, height);
 var gfx = Graphics.FromImage(bmp);
 
 gfx.FillRectangle(Brushes.Blue, new RectangleF(0, 0, width, height));

 foreach (var r in rectangles)
 {
  gfx.DrawRectangle(Pens.Black, 
   new Rectangle(r.X, r.Y, r.Width - 1, r.Height - 1));

  gfx.DrawString(r.Slice.Elements.First().Object.ToString(), font, 
   Brushes.White, r.X, r.Y);
 }

 var form = new Form() { AutoSize = true };
 form.Controls.Add(new PictureBox()
  { Width = width, Height = height, Image = bmp });
 form.ShowDialog();
}

And finally to generate a Treemap in LinqPad
void Main()
{
 const int Width = 400;
 const int Height = 300;
 const double MinSliceRatio = 0.35;

 var elements = new[] { 24, 45, 32, 87, 34, 58, 10, 4, 5, 9, 52, 34 }
  .Select (x => new Element<string> { Object = x.ToString(), Value = x })
  .OrderByDescending (x => x.Value)
  .ToList();

 var slice = GetSlice(elements, 1, MinSliceRatio).Dump("Slices");
 
 var rectangles = GetRectangles(slice, Width, Height)
  .ToList().Dump("Rectangles");
 
 DrawTreemap(rectangles, Width, Height);
}

References:

Wednesday, August 28, 2013

Windows Azure Caching and transient faults

When using remote services over the wire we should always plan for transient failures. Windows Azure Caching like any services in an Azure world is prone to such problem. Out of the box making calls to the cache server will fail from time to time due to network issues. Typically you will get those kind of exceptions:
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later.
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out.
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server.

For that reason it is a best practice to implement some kind of retry logic around your code calling the cache server. We could have used the Transient Application Block to manage that. But a few months ago I found somewhere that from time to time the DataCache object lose it's internal connection to the cache server. A simple way to fix this is to re-create a DataCache instance and retry the operation.

In the implementation below I'm keeping a reference to the DataCacheFactory and DataCache objects (another best practice). The CreateDataCache factory method will come handy later.

public class CachingService
{
    private DataCacheFactory cacheFactory;
    private DataCache cache;
   
    private DataCache Cache
    {
        get
        {
            if (this.cache == null)
            {
                this.CreateDataCache();
            }

            return this.cache;
        }
    }

    private void CreateDataCache()
    {
        this.cacheFactory = new DataCacheFactory();
        this.cache = this.cacheFactory.GetDefaultCache();
    }

    // ...
}

Then I have this SafeCallFunction I use whenever I want to work with the DataCache object. Notice that the only thing I do to retry the operation is to call the factory method to re-create the DataCache object.

private object SafeCallFunction(Func<object> function)
{
    try
    {
        return function.Invoke();
    }
    catch (DataCacheException)
    {
        // Retry by first re-creating the DataCache
        try
        {
            this.CreateDataCache();
            return function.Invoke();
        }
        catch (DataCacheException)
        {
            // Log error
        }
    }

    return null;
}

Finally in the rest of the class I can use the SafeCallFunction like this
public object CacheGet(string key)
{
    return this.SafeCallFunction(() => this.Cache.Get(key));
}

public void CachePut(string key, object cacheObject)
{
    this.SafeCallFunction(() => this.Cache.Put(key, cacheObject));
}

public void CacheRemove(string key)
{
    this.SafeCallFunction(() => this.Cache.Remove(key));
}

So far after a few weeks of using this implementation the single retry never failed on us. Before that we had around 5-10 failures daily for about 500k calls to the cache server. I would still recommend using a more robust retry policy with Windows Azure Caching but I think it's interesting to know that simply instantiating a new DataCache can fix most failures.

References

Caching in Windows Azure
Best Practices for using Windows Azure Cache
Optimization Guidance for Windows Azure Caching
The Transient Fault Handling Application Block

Wednesday, July 31, 2013

Using Windows Azure Caching efficiently across multiple Cloud Service roles

Windows Azure Caching is a great way to improve performance of your Azure application at no additional cost. The cache is running alongside your application in Cloud Service roles. The only thing you need to decide is how much memory of the role you want use for it (for co-located cache role). You can also dedicate the all memory of a role to caching if you want (with dedicated cache role). Roles that host caching are called cache clusters.

Starting with Azure Caching is so easy that it can be a while before you fully understand the best way to use it. On a recent project my first tough was to enable caching on all the Cloud Service roles as co-located service. This was causing us problems.

First of all, been a developer I debug my application using the local Azure compute emulator. The emulator runs one cache service for each instances of roles with cache clusters. The application has 2 web roles and 1 worker role so when I start a debugging session with multiple instances per role I need a lot of memory to run everything. More importantly, cache clusters do not share cached data between each other. This caused us to have stale data in the application.

That is when I figured out that I needed to read a bit more on Azure Caching if I was to use it efficiently.

Understanding cache clusters


When you enable caching on an Azure role each instance of that role will run a cache service using a portion of the memory (or all of it if it's a dedicated cache role). The cache services running on each instance of a single role are managed as a single cache cluster. Cache services can talk to each other and synchronize data but only inside the same cache cluster (same role). That is why enabling caching of many roles might not be the best thing to do.


Another thing to mention is that cache clusters can only be created on small role instances or bigger. The reason is that with extra small instance you only get 768MB of RAM which is pretty much all used up by anything you run on those instances.

Now the enable a cache cluster on your role go to the role property page on the Caching tab.


Here you will notice that I also enabled notifications which will allow us to efficiently use local caches later.

For more information on the different configuration options for cache clusters go here.

Configuring roles to use cache clients


Now that we took care of the server side of caching configuration let's talk about the client side. Each instances of each roles inside the same cloud deployment can connect to a cache cluster. If you run only one cluster then you are guarantied to access the same cached data from whatever role you are inside your application (as long you have a valid configuration).


One nice feature we can enable in each role configuration is the local cache client. With this we can cache data locally in a role instance memory the data we recently fetched from the cache cluster for even faster access. Remember the Notification option we enabled on the server side? Using the configuration below in the Web.config or App.config of your role will ensure data stored in the local cache client gets updated whenever the cache server version of that data changes. Basically, the local cache client will invalidate data based on notifications received from the cache cluster.


For more information on client side configuration go here.

Other concerns


This post is only about an overview of the consideration of running multiple clusters versus a single one. Using Azure Caching there are a lot more configuration options you need to take a look at here. Also really important is how to use Azure Caching in your application.

Conclusions


I've spend a lot of time figuring out how all of this was working. I hope this post will help you with your learning experience.

Other useful links




Monday, July 22, 2013

Handling Azure Storage Queue poison messages

This post will talk about what to do now that we are handling poison messages in our Azure Storage Queues.

First, let's review what we'll done so far.


Messages that continuously fail to process will end up in the Error Queue. Someone asked me why do we need error queues at all? We could simply log the errors and delete the message right? Well, if you have a really efficient and pro-active DevOps team I suppose logging errors along with the original messages ought to be enough.
Someone will review why the message failed and if it was only a transient error then he could send the original message again in the queue.

We could also store failed messages into an Azure Storage Table.
Then we could simply monitor new entries in this table and act on it. Again, the original message should be stored in the table so we could send it again if we choose.

I think the best reason to use a queue for error message is if you want to have an administrative tools to monitor, review and re-send messages. In this case the queue mechanics let's you handle those messages like any other process using queues do.

For me one of the unpleasant side effect of using error queues is that all the queues in my system are now multiplied by two (one normal and one error queue). It's not too bad if your naming scheme is consistent but even then if you do operational work using a tool like Cerebrata Azure Management Studio or even from Visual Studio's Server Explorer you will feel overwhelmed by the quantity of queues.

Managing queue messages with Cerebrata Azure Management Studio
Managing queue messages with Visual Studio Server Explorer

Whatever you do, I suggest you always at least log failures properly.  Later while debugging the issue you will be thankful to easily match the failure logs with the message who caused it.