tag:blogger.com,1999:blog-57249583970095841472024-03-13T10:44:27.351-04:00The Never Ending JourneyAnonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.comBlogger42125tag:blogger.com,1999:blog-5724958397009584147.post-76295357219627808582015-01-05T13:02:00.000-05:002015-01-05T13:02:06.768-05:00Unit testing smells and best practicesOne of the most difficult task I know as a software engineer is to write good unit test. Of course multi-threading and distributed systems are also challenging but in this post I want to share some of the good and bad things I've learning doing unit testing over the last decade.<br />
<br />
In a series of posts I'll cover mistakes I made in the past and good techniques I learned along the way. Come back often as this list will grow over time.<br />
<br />
<h3>
The smells</h3>
<ul>
<li><a href="http://pascallaurin42.blogspot.com/2014/09/unit-test-smells-non-public-class.html">The non-public class</a></li>
<li><a href="http://pascallaurin42.blogspot.com/2014/10/unit-testing-smells-class-constructor.html">The class constructor is not easy to call</a></li>
<li><a href="http://pascallaurin42.blogspot.com/2015/01/unit-testing-smells-method-is-not-public.html">The method is not public</a></li>
</ul>
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-4333024239345084402015-01-04T15:20:00.000-05:002015-01-04T15:37:16.296-05:00Unit testing smells: The method is not publicA recurring pattern emerge often when unit testing, you find an interesting method you want to test but that method is private. Of course you could call that method through other public methods on the class but it might not be easy to cover all cases. This usually is a design smell for a <a href="http://pascallaurin42.blogspot.com/2011/04/single-responsibility-principle.html">Single Responsibility Principle</a> violation. More on this later.<br />
<br />
For now, to properly test the private method we have the following options:<br />
<ul>
<li>Use reflection</li>
<li>Change the method visibility to public or internal</li>
<li>Extract the method to another class</li>
</ul>
<br />
Again, if you previously read my other post on <a href="http://pascallaurin42.blogspot.com/2014/09/unit-test-smells-non-public-class.html">non public class</a> and <a href="http://pascallaurin42.blogspot.com/2014/10/unit-testing-smells-class-constructor.html">inaccessible constructor</a> you know you should avoid using reflection in tests.<br />
<br />
Let's take a look at the other options<br />
<br />
<h3>
Change the method visibility to public or internal</h3>
The first thing we should do is to challenge why that method is private in the first place. Why can't we just change its visibility to public or internal (and <a href="http://pascallaurin42.blogspot.com/2014/09/unit-test-smells-non-public-class.html">use the <code>InternalsVisibleTo</code> attribute</a>)?<br />
<br />
This breaks encapsulation of the class and will let developers call the method directly in production code. That might not be what you intended to do but can help you in the short term. Abusing this technique will burn you in the long run so be careful with it.<br />
<br />
Fortunately, there is a simpler way: extract the method to another class. <br />
<br />
<h3>
Extract the method to another class</h3>
If you are lucky, the method you want to extract is static and don't call any other method on the original class. You should be able to extract that method to a static utility class.<br />
<br />
But what to do when the method calls other methods and use fields or properties on the class? For that we must analyze the class to extract responsibilities using the <a href="http://pascallaurin42.blogspot.com/2011/04/single-responsibility-principle.html">Single Responsibility Principle</a>.<br />
<br />
<h3>
Single Responsibility Principle</h3>
Can you tell what is the purpose of your class in one sentence without using words like: <em>and</em>, <em>but</em>, <em>also</em>, <em>or</em>, etc.? If not then your class is doing more than one thing. Each segments of the sentence could be in different classes that only have one responsibility each. If the method you want to extract use the same fields than other methods you may want to push them by parameter instead before extraction to the other class.<br />
<br />
Find more about the Single Responsibility Principle in an <a href="http://pascallaurin42.blogspot.com/2011/04/single-responsibility-principle.html">old post of mine</a>.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-42403147758011571492014-10-04T15:04:00.000-04:002014-10-04T15:04:08.600-04:00Unit testing smells: The class constructor is not easy to callWhen we want to test an instance method on a class the first challenge is to create an instance of that class. Hopefully the class constructor is easy to call but that's not always the case.<br />
<br />
Let's review a few cases where the constructor might prevent us to write tests easily:<br />
<ul><li>The constructor is private or internal </li>
<li>It requires too many arguments or it is too hard to create/provide those arguments</li>
<li>The constructor calls external dependencies</li>
</ul><br />
<h3>The case of the internal or private constructor</h3>For this we can follow the same advices than with <a href="http://pascallaurin42.blogspot.com/2014/09/unit-test-smells-non-public-class.html">testing non-public class</a>. But whatever you do please don't use reflection to call the constructor.<br />
<br />
Also stated in <a href="http://pascallaurin42.blogspot.com/2014/09/unit-test-smells-non-public-class.html">testing non-public class</a>, you should ask yourself if it would be better to test this class via another public class that use it. A private constructor usually means that there is a factory somewhere you should use to instantiate the class. You should figure out a way to use that factory or redesign it in case you have difficulties to work with it in unit tests.<br />
<br />
When dealing with internal access modifier I usually use the <code>InternalVisibleTo</code> attribute for my test project. <br />
<br />
<pre class="brush: csharp">[assembly: InternalsVisibleTo("Contoso.MyApp.UnitTests.dll")]
</pre><br />
<h3>The case of the difficult constructor arguments</h3>If the problem is that the constructor requires too many arguments then you should ask yourself if this class is too big, maybe it has too many responsibilities and don't follow the <a href="http://pascallaurin42.blogspot.com/2011/04/single-responsibility-principle.html">Single Responsibility Principle</a>? In that case the class should be splitted into two or more classes. This way it will be easier to test the smaller class because its constructor should require less arguments.<br />
<br />
Another case is when the arguments are difficult to create or provide. Maybe that in order to create one object to pass as parameter we need to create yet more objects. Micheal Feathers calls it the Onion Parameter in his book <a href="http://amzn.com/0131177052">Working Effectively with Legacy Code</a>. In that case I usually invest into a bit of testing infrastructure like <a href="http://www.dofactory.com/Patterns/PatternFactory.aspx">factory methods</a> or <a href="http://www.natpryce.com/articles/000714.html">Test Data Builder</a> in order to help me write tests faster afterwards. <br />
<br />
Finally some parameters could be classes that wrap services or external dependencies. In that case I abstract the dependency using an interface and the <a href="http://en.wikipedia.org/wiki/Inversion_of_control">Inversion of Control Principle</a>. First extract an interface from the class. Next change the constructor to use this interface instead of the class. Finally use a mocking framework like <a href="https://github.com/Moq/moq4">Moq</a> to create a fake (or create your own by hand if you don't like mocking frameworks) and pass it to the constructor.<br />
<br />
<pre class="brush: csharp">public class MyService
{
// public MyService(SomeDataAccessLayer dal, SomeExternalService externalService)
public MyService(ISomeDataAccessLayer dal, ISomeExternalService externalService)
{
// ...
}
// ...
}
</pre><br />
<h3>The case of the external dependencies calls</h3>When the constructor itself calls an external dependency I usually refactor the constructor to inject this dependency via a parameter instead, this is called <a href="http://en.wikipedia.org/wiki/Dependency_injection">dependency injection</a> and usually comes with the <a href="http://en.wikipedia.org/wiki/Inversion_of_control">Inversion of Control Principle</a>. This is also a good occasion to take a look at a IoC Container like <a href="https://unity.codeplex.com/">Unity</a>.<br />
<br />
<pre class="brush: csharp">public class MyService
{
// public MyService()
public MyService(ISomeExternalService externalService)
{
// ...
// var externalService = new ExternalService();
externalService.CallService();
}
// ...
}
</pre>Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com3tag:blogger.com,1999:blog-5724958397009584147.post-78478158409866976972014-09-06T19:44:00.000-04:002014-09-06T19:44:48.923-04:00Unit test smells: The non-public classWriting unit tests for a method on a class that is not public is doable but not straight forward. It could be done using a bit of reflection like this<br />
<br />
<pre class="brush: csharp">var type = Type.GetType("MyProject.MyClass");
var methodInfo = type.GetMethod("TheMethod");
var classInstance = Activator.CreateInstance(type, null);
methodInfo.Invoke(classInstance, null);
</pre><br />
That is quite a bit of work. Of course you could create some test infrastructure to avoid duplication in every tests or you may find libraries online for that. But still, writing tests like this smells funny to me.<br />
<br />
In short, <strong><em>you should never write tests directly against a non-public class.</em></strong><br />
Let's take a look at the 2 possible cases for this: the <strong>private class</strong> and the <strong>internal class</strong>.<br />
<br />
<h3>The case of the private class</h3>For a class to be private means it is nested inside another class<br />
<br />
<pre class="brush: csharp">public class A
{
// ...
private class B
{
// ...
}
}
</pre><br />
In a situation like this <code>class B</code> can only be used by <code>class A</code>. Anything <code>class B</code> do is only for serving <code>class A</code>. If you want to test a method in <code>class B</code> you should find out how <code>class A</code> use <code>class B</code> and write your tests against <code>class A</code> public API.<br />
<br />
If you find there is no way to reach the method you want to test in <code>class B</code> through <code>class A</code> it simply means that you just found dead code! If you found a way but find it too hard to setup a test then maybe <code>class B</code> is not simply a private utility class. In that case I would extract <code>class B</code> from inside <code>class A</code> and change its access modifier to <code>internal</code>. Unfortunately, now any class inside the same assembly could use <code>class B</code>. It is a trade-off I'm willing to pay because this case is really rare and C# still lack a proper access modifier scoped to the current namespace.<br />
<br />
<h3>The case of the internal class</h3>Internal class could also be tested through reflection. Personally I prefer to use the <code>InternalsVisibleToAttribute</code> and give access to internals types to my unit test project. To do it you need to add the attribute to the project under test <code>AssemblyInfo.cs</code> file like this<br />
<br />
<pre class="brush: csharp">[assembly: InternalsVisibleTo("Contoso.MyApp.UnitTests.dll")]
</pre><br />
If your project is signed you will need to also provide the assembly's public key like this<br />
<br />
<pre class="brush: csharp">[assembly: InternalsVisibleTo("Contoso.MyApp.UnitTests.dll, PublicKey=1234...789")]
</pre><br />
Again, this is a trade-off: you give special access to your internal types only for testing but it is <strong><em>way</em></strong> better than changing the class access modifier to <code>public</code>. <strong><em>You should never change a type access modifier to <code>public</code> for testing!</em></strong><br />
<br />
<h3>Conclusion</h3>I always try to write tests against <code>public</code> classes and methods. Sometime an internal class will grow to be very complex over time and become its own new component. In a solution were we create a lot of small projects we would simply create a new one for such component and expose it publicly but that is not what I do. I usually try to create the minimal number of projects in my solution so testing <code>internal</code> classes using the <code>InternalsVisibleToAttribute</code> is a good trade-off for me. Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2tag:blogger.com,1999:blog-5724958397009584147.post-17631186657367591622014-07-13T16:36:00.000-04:002014-07-13T16:36:13.092-04:00Exploring BDDfyBDDfy is a <a href="http://dannorth.net/introducing-bdd/">BDD</a> library (part of the larger <a href="http://docs.teststack.net/">TestStack project</a>). It's function is to turn regular unit tests to BDD style tests (using the <a href="http://docs.behat.org/guides/1.gherkin.html">Gherkin syntax</a>). You can read more about BDDfy <a href="http://docs.teststack.net/bddfy/index.html">here</a>.<br />
<br />
BDDfy can be used with any test framework or runner.<br />
<br />
<h2>Acquiring</h2>To add BDDfy to your test project via NuGet run this command in the Package Manage Console.<br />
<br />
<pre><code>Install-Package TestStack.BDDfy
</code></pre><br />
Optionally, you can also install code samples with this package.<br />
<br />
<pre><code>Install-Package TestStack.BDDfy.Samples
</code></pre><br />
<h2>Exploring</h2><br />
<h3>Hello world</h3>For this part I'll be using the xUnit test framework. Let's start with something simple.<br />
<br />
<pre class="brush: csharp">public class FirstTest
{
void GivenTwoAndTwo()
{
// ...
}
void WhenIAddThem()
{
// ...
}
void ThenTheAnwserShouldBe4()
{
// ...
}
[Fact]
public void ExecuteFirstTest()
{
this.BDDfy();
}
}
</pre><br />
All the magic is done by the <code>BDDFy</code> extension methods. This will scan the <code>FirstTest</code> class for methods starting with keywords like <code>Given</code>, <code>When</code> and <code>Then</code>. Next, it will run the methods in order (BDD style). Finally, we will get a nice output report like this.<br />
<br />
<a href="http://3.bp.blogspot.com/-p_ZqTr3PA2k/U8LrDNj2nrI/AAAAAAAAC9I/Go_2_yakQak/s1600/BDDfy1.png" imageanchor="1"><img alt="Test output in Visual Studio" border="0" src="http://3.bp.blogspot.com/-p_ZqTr3PA2k/U8LrDNj2nrI/AAAAAAAAC9I/Go_2_yakQak/s320/BDDfy1.png" title="" /></a><br />
<br />
BDDfy will also generate a <code>BDDfy.html</code> file in the test project output folder. This is the report of all BDDFyed tests.<br />
<br />
<a href="http://2.bp.blogspot.com/-GtxsuOdOoFc/U8LrJDMkkYI/AAAAAAAAC9Q/UFX0eNAlJtA/s1600/BDDfy2.png" imageanchor="1"><img alt="HTML tests report" border="0" src="http://2.bp.blogspot.com/-GtxsuOdOoFc/U8LrJDMkkYI/AAAAAAAAC9Q/UFX0eNAlJtA/s1600/BDDfy2.png" title="" /></a><br />
<br />
<h3>Using attributes to customize the test</h3>BDDfy follow conventions when scanning a class for methods of interest, you can find a list <a href="http://docs.teststack.net/BDDfy/Usage/MethodNameConventions.html">here</a>. If we need more control we can do it by using attributes.<br />
<br />
<pre class="brush: csharp">[Story(
AsA = "As someone lazy",
IWant = "I want the computer to add 2 number",
SoThat = "I don't have to do the math myself")]
public class TestWithAttributesToOverriteText
{
[Given("Given 2 + 2")]
void GivenTwoAndTwo()
{
// ...
}
[When(" + ")]
void WhenIAddThem()
{
// ...
}
[Then("Then the anwser = 4")]
void ThenTheAnwserShouldBe4()
{
// ...
}
void AndThenItShouldDisplayTheAnwser()
{
}
[Fact]
public void ExecuteTestWithAttributes()
{
this.BDDfy();
}
}
</pre><br />
First, the <code>[Story]</code> attribute allow us to provide the classic story definition for the test. Other attributes like <code>[Given]</code>, <code>[When]</code> and <code>[Then]</code> allow us to provide a custom description for the steps. Also, using the attributes will allow us to name the step methods the way we want. <br />
<br />
<h3>Creating more than one scenario per story</h3>Usually a story contains more than one test or scenario. We can do this using nested classes.<br />
<br />
<pre class="brush: csharp">[Story(
Title = "Using story attribute and setting the Title!",
AsA = "As someone learning BDDfy",
IWant = "I want to try splitting scenario in separated classes",
SoThat = "My code is cleaner")]
public class TestWithStoryAndScenarioInSeparatedClasses
{
[Fact]
public void FirstScenario()
{
new S1().BDDfy("Custom scenario title");
}
[Fact]
public void SecondScenario()
{
new S2().BDDfy();
}
private class S1
{
void GivenWhatever() { // ... }
void WhenSomethingHappens() { // ... }
void ThenProfit() { // ... }
}
private class S2
{
void GivenWhatever() { // ... }
void WhenSomethingElseHappens() { // ... }
void ThenProfit() { // ... }
}
}
</pre><br />
This will group scenarios together in the output report.<br />
<br />
<h3>Using the fluent API for even more control</h3>With what we've seen previously we need to create a new test class for each scenario we have. That leads to a lot of duplicated code unless we delegate to a common test fixture. An alternative is to use BDDfy fluent API to get some code reuse between our scenarios.<br />
<br />
<pre class="brush: csharp">public class TestWithFluentApi
{
[Fact]
public void ReusingStepForScenario1()
{
new TestWithFluentApi()
.Given(s => s.GivenWhatever(), "Given some pre-condition")
.And(s => s.AndOtherGiven(54))
.When(s => s.WhenSomethingElseHappens())
.Then(s => s.ThenProfit())
.And(s => s.AndManyMore(45))
.BDDfy();
}
[Fact]
public void ReusingStepForScenario2()
{
new TestWithFluentApi()
.Given(s => s.GivenWhatever(), "Given some pre-condition")
.And(s => s.AndOtherGiven(123))
.When(s => s.WhenSomethingElseHappens())
.Then(s => s.ThenProfit())
.And(s => s.AndManyMore(321), "And {0} more things!")
.BDDfy("Scenario 2 with steps re-use");
}
void GivenWhatever() { // ... }
void AndOtherGiven(int input) { // ... }
void WhenSomethingElseHappens() { // ... }
void ThenProfit() { // ... }
void AndManyMore(int expected) { // ... }
}
</pre><br />
With this style of test, not only we can reuse steps between scenarios we also gain the ability to parameterize the steps.<br />
<br />
<h2>Assessment</h2>I've only scratched the surface of what BDDfy can do. You can read more about on BDDfy <a href="http://docs.teststack.net/bddfy/usage/index.html">usage</a> and <a href="http://docs.teststack.net/bddfy/customizing/index.html">customization</a> on the project web site.<br />
<br />
In the past I've use <a href="http://www.specflow.org/">SpecFlow</a> for my BDD tests. With SpecFlow you write your specification in a text file using the Gherkin language. The tool then parse the file and execute corresponding method for each steps. Having a text file seems interesting because we could have a business analyst write those. In really, developers ends up writing the stories and scenarios anyway.<br />
<br />
This is why I like BDDfy, it's easy to learn and gives developers a lot of control over the way we create BDD style tests. <br />
<br />
One more for my toolbox!<br />
<br />
I hope you enjoyed this introduction to BDDfy.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-28636101883920232882014-06-22T22:41:00.000-04:002014-06-22T22:41:59.032-04:00Visualizing Nuget packages dependencies without Visual Studio UltimateIn <a href="http://pascallaurin42.blogspot.com/2014/05/managing-nuget-packages-dependencies.html">my previous post I've shown the Package Visualizer tool</a>. Unfortunately, it's only available in the Ultimate version of Visual Studio. But all is not lost because even with a Pro version with can open <a href="http://en.wikipedia.org/wiki/DGML">DGML</a> files.<br />
<br />
I've created a <a href="http://www.linqpad.net/">LinqPad</a> query that analyse packages.config files and create a DGML diagram like Package Visualizer does. I've also added things like GAC libraries and normal file based library to the mix. You can get the full <a href="https://gist.github.com/plaurin/b4bc53428f01dc722afb">Gist here</a>. Now let's take a look at some code…<br />
<br />
<h3>Main</h3>Here we set a few options for our query: some file extensions to ignore when scanning for projects and more importantly the root folder path to start scanning for project files.<br />
<br />
<pre class="brush: csharp">private string[] projectExtensionExclusions = new[] { ".vdproj", ".ndproj" };
private string rootFolder = @"C:\Users\Pascal\Dev\MyProject";
void Main()
{
LoadAllProjects();
LoadAllPackagesConfig();
GenerateDGML(Path.Combine(rootFolder, "Dependencies.dgml"));
}
</pre><br />
<h3>Data structures to uses</h3>Then we define some fields and basic classes to help us gather the information<br />
<br />
<pre class="brush: csharp">private List<Project> projects = new List<Project>();
private List<Package> packages = new List<Package>();
private List<Library> libraries = new List<Library>();
public class Project
{
public Project()
{
this.Projects = new List<Project>();
this.Libraries = new List<Library>();
this.Packages = new List<Package>();
}
public string Path { get; set; }
public string Name { get; set; }
public List<Project> Projects { get; private set; }
public List<Library> Libraries { get; private set; }
public List<Package> Packages { get; private set; }
}
public class Package
{
public string Name { get; set; }
public string Version { get; set; }
}
public class Library
{
public string Name { get; set; }
public bool IsGAC { get; set; }
}
</pre><br />
<h3>LoadAllProjects</h3>Now we can start scanning for projects to load. Next we open each project files and extract all dependencies like other project, a local library or a GAC reference. We keep all this info in the project instances for later.<br />
<br />
<pre class="brush: csharp">private void LoadAllProjects()
{
XNamespace ns = "http://schemas.microsoft.com/developer/msbuild/2003";
var projectFiles = Directory.GetFiles(rootFolder, "*.*proj",
SearchOption.AllDirectories)
.Where (pf => !projectExtensionExclusions.Any(ex => pf.EndsWith(ex)));
foreach (var pf in projectFiles)
this.projects.Add(
new Project { Path = pf, Name = Path.GetFileNameWithoutExtension(pf) });
// Get all projects, local libraries and GAC references
foreach (var project in this.projects)
{
var projectDoc = XDocument.Load(project.Path);
foreach (var pr in projectDoc.Descendants(ns + "ProjectReference"))
{
var prj = projects.SingleOrDefault(p =>
p.Name == pr.Element(ns + "Name").Value);
if (prj != null)
project.Projects.Add(prj);
else
(pr.Element(ns + "Name").Value
+ " project reference not found in file " + project.Path).Dump();
}
foreach (var r in projectDoc.Descendants(ns + "Reference")
.Where (r => !r.Value.Contains(@"\packages\")))
project.Libraries.Add(GetOrCreateLibrary(
r.Attribute("Include").Value, !r.Elements(ns + "HintPath").Any()));
}
}
</pre><br />
<h3>LoadAllPackagesConfig</h3>Finally we scan for packages.config files, the ones responsible for maintaining the NuGet packages dependencies for a project. Again we extract the dependencies information from the files and keep it for later.<br />
<br />
<pre class="brush: csharp">private void LoadAllPackagesConfig()
{
foreach (var pk in Directory.GetFiles(rootFolder, "packages.config",
SearchOption.AllDirectories)
.Where (pc => !pc.Contains(".nuget")))
{
var project = this.projects.SingleOrDefault(p =>
Path.GetDirectoryName(p.Path) == Path.GetDirectoryName(pk));
if (project == null)
("Project not found in same folder than package " + pk).Dump();
else
{
foreach (var pr in XDocument.Load(pk).Descendants("package"))
{
var package = GetOrCreatePackage(
pr.Attribute("id").Value, pr.Attribute("version").Value);
project.Packages.Add(package);
}
}
}
}
</pre><br />
<h3>GenerateDGML</h3>Here we generate the final DGML file which is simply an XML file. The <a href="http://schemas.microsoft.com/vs/2009/dgml/">schema</a> is quite simple: a root element DirectedGraph, a Nodes section and a Links section, all of which are mandatory. We also add a Styles section to colorize the different kind of nodes: projects, packages, libraries and GAC libraries.<br />
<br />
<pre class="brush: csharp">private XNamespace dgmlns = "http://schemas.microsoft.com/vs/2009/dgml";
private void GenerateDGML(string filename)
{
var graph = new XElement(dgmlns + "DirectedGraph",
new XAttribute("GraphDirection", "LeftToRight"),
new XElement(dgmlns + "Nodes",
this.projects.Select (p => CreateNode(p.Name, "Project")),
this.libraries.Select (l => CreateNode(l.Name,
l.IsGAC ? "GAC Library" : "Library", l.Name.Split(',')[0])),
this.packages.Select (p => CreateNode(p.Name + " " + p.Version, "Package")),
CreateNode("AllProjects", "Project", label: "All Projects", @group: "Expanded"),
CreateNode("AllPackages", "Package", label: "All Packages", @group: "Expanded"),
CreateNode("LocalLibraries", "Library", label: "Local Libraries", @group: "Expanded"),
CreateNode("GlobalAssemblyCache", "GAC Library", label: "Global Assembly Cache", @group: "Collapsed")),
new XElement(dgmlns + "Links",
this.projects.SelectMany(p => p.Projects.Select(pr => new { Source = p, Target = pr } ))
.Select (l => CreateLink(l.Source.Name, l.Target.Name, "Project Reference")),
this.projects.SelectMany(p => p.Libraries.Select(l => new { Source = p, Target = l } ))
.Select (l => CreateLink(l.Source.Name, l.Target.Name, "Library Reference")),
this.projects.SelectMany(p => p.Packages.Select(pa => new { Source = p, Target = pa } ))
.Select (l => CreateLink(l.Source.Name, l.Target.Name + " " + l.Target.Version, "Installed Package")),
this.projects.Select (p => CreateLink("AllProjects", p.Name, "Contains")),
this.packages.Select (p => CreateLink("AllPackages", p.Name + " " + p.Version, "Contains")),
this.libraries.Where (l => !l.IsGAC).Select (l => CreateLink("LocalLibraries", l.Name, "Contains")),
this.libraries.Where (l => l.IsGAC).Select (l => CreateLink("GlobalAssemblyCache", l.Name, "Contains"))),
// No need to declare Categories, auto generated
new XElement(dgmlns + "Styles",
CreateStyle("Project", "Blue"),
CreateStyle("Package", "Purple"),
CreateStyle("Library", "Green"),
CreateStyle("GAC Library", "LightGreen")));
var doc = new XDocument(graph);
doc.Save(filename);
}
</pre><br />
<h3>Conclusion</h3>All that is left is to open the Dependencies.dgml file in Visual Studio<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-MKGebRYEg28/U6Ohn99oLVI/AAAAAAAAC80/EWLeOLFnGNc/s1600/nvq1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-MKGebRYEg28/U6Ohn99oLVI/AAAAAAAAC80/EWLeOLFnGNc/s1600/nvq1.png" /></a></div><br />
I've left a few utility methods out of the inline code in this post but you can get all the code from the <a href="https://gist.github.com/plaurin/b4bc53428f01dc722afb">Gist</a>. Feel free to grab a copy of the file and adapt it to your heart's content. It would be easy to create a small Console Application and call it from command line if you don't like LinqPad.<br />
<br />
There is still a lot more I could add to the query like extracting projects and library versions from the DLL, dependencies between NuGet packages from .nupkg files and highlighting duplicates NuGet packages with different version. Still, it's enough for me in it's current form.<br />
<br />
I hope this will help you figure out your NuGet packages usage and dependencies in your solution.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com5tag:blogger.com,1999:blog-5724958397009584147.post-846403705274230342014-05-24T15:20:00.000-04:002014-05-24T15:41:48.939-04:00Managing NuGet packages dependencies with the Package Visualizer tool<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If you
ever used <a href="http://www.nuget.org/">NuGet</a> on a large enough solution
you know you can get into trouble when projects reference different versions of the same NuGet
package. That happens a lot in Azure projects as the libraries/packages get
updated all the time.</div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I'm
really surprise when I talk to people using NuGet everyday that they don't know
about the <a href="http://docs.nuget.org/docs/workflows/package-visualizer">Package
Visualizer</a> tool. <span style="font-style: italic;">(<b>update:</b> I've been told
that this feature requires VS Ultimate and is not available in the Pro version.
I'm still going to show it to you but stay tune for another post with a free
alternative later)</span></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
NuGet Package Visualizer in Visual Studio</h2>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Once you open
up a solution in Visual Studio you can go to the <span style="font-weight: bold;">Tools</span>
menu, <span style="font-weight: bold;">NuGet Package Manager</span> and <span style="font-weight: bold;">Package Visualizer</span>.</div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-DO6xhiOFt5k/U4DtrCTxVNI/AAAAAAAAC8A/6uQ0179tTq0/s1600/npv1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-DO6xhiOFt5k/U4DtrCTxVNI/AAAAAAAAC8A/6uQ0179tTq0/s1600/npv1.png" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">This will
analyse all the </span><span style="font-size: 11pt; font-weight: bold;">packages.config</span><span style="font-size: 11pt;"> files in
the solution and generate a </span><a href="http://en.wikipedia.org/wiki/DGML" style="font-size: 11pt;">DGML</a><span style="font-size: 11pt;">
diagram of all NuGet packages and</span><span style="font-size: 11pt;"> </span><span style="font-size: 11pt;">projects of the solution.</span><span style="font-size: 11pt;"> </span><span style="font-size: 11pt;">The
diagram will help us see packages usage in the solution and find the ones with
different versions.</span><span style="font-size: 11pt;"> </span><span style="font-size: 11pt;">Below you can see
that I've tried this on the </span><a href="https://roslyn.codeplex.com/" style="font-size: 11pt;">Roslyn (open
source C# compiler)</a><span style="font-size: 11pt;"> solution.</span></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-EPRgepo0oWQ/U4DtrBwjYQI/AAAAAAAAC8I/9jL0DD5jdbQ/s1600/npv2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://1.bp.blogspot.com/-EPRgepo0oWQ/U4DtrBwjYQI/AAAAAAAAC8I/9jL0DD5jdbQ/s1600/npv2.png" /></a></div>
<div style="margin: 0in;">
</div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The first
thing to note (and a surprise to me!) is that Roslyn use the <a href="https://github.com/xunit/xunit" target="">XUnit testing framework</a> and not MsUnit! More seriously we can quickly see that we have no duplicate
packages with different versions. If we
compare that to this sample solution I created we can see I'm using two
versions of the Json.NET library. Now I know I should update the ClassLibrary1
project to use the new version of the package.</div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Upd0mRMfzKY/U4DtrAssN9I/AAAAAAAAC8M/8HTUSCn4pMU/s1600/npv3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-Upd0mRMfzKY/U4DtrAssN9I/AAAAAAAAC8M/8HTUSCn4pMU/s1600/npv3.png" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div lang="fr-CA" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">Of
course, this only work for NuGet packages but it would be useful to have
something like this for regular DLL references.</span><span style="font-size: 11pt;">
</span><span style="font-size: 11pt;">I'll try to work on a LinqPad query to generate such a DGML graph with
all projects, libraries and packages.</span><span style="font-size: 11pt;">
</span><span style="font-size: 11pt;">Stay tune till next time.</span></div>
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2tag:blogger.com,1999:blog-5724958397009584147.post-29300607249901895592014-04-25T15:49:00.000-04:002014-04-25T15:49:03.038-04:00Looking inside a NuGet package with NuGet Package ExplorerWhen I really want to learn something new (like a new tool, technology or a programming language) I do two things<br />
<br />
<ul>
<li>I try it myself</li>
<li>Check out what others have done</li>
</ul>
<br />
I'm currently learning how to create my own NuGet packages so I'm trying to do a lot of things on my own, but I would also like to see how existing packages are made.<br />
<br />
After installing a NuGet package in your project you can go in the <b>packages</b> folder and unzip the <b>.nupkg</b> file (yes, it's only a zip file with a different extension). Fortunately, there is an easier. <a href="http://npe.codeplex.com/" target="_blank">NuGet Package Explorer</a> is an open source tool available on CodePlex.<br />
<br />
With it you can load a package from the official NuGet feed or any other feeds you want even local feeds.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-pP2RWtqhGgQ/U1q5fndGVCI/AAAAAAAAC64/FcZ6BvglH24/s1600/NPE2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-pP2RWtqhGgQ/U1q5fndGVCI/AAAAAAAAC64/FcZ6BvglH24/s1600/NPE2.png" /></a></div>
<br />
Then when we open a package we can explore its content and even go inside individual files<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-Jxdg52Ki_cw/U1q5sndF5RI/AAAAAAAAC7M/buX04lgpo18/s1600/NPE4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-Jxdg52Ki_cw/U1q5sndF5RI/AAAAAAAAC7M/buX04lgpo18/s1600/NPE4.png" /></a></div>
<br />
<br />
NuGet Package Explorer also allow us to edit files and the package itself if we want.<br />
<br />
One trick I like to do is to add my local NuGet package download cache as a feed. To do that in the Tools menu select View NuGet download cache.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-yVjJFpttcN4/U1q5p7EAj8I/AAAAAAAAC7E/ATe-UACI0xE/s1600/NPE1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-yVjJFpttcN4/U1q5p7EAj8I/AAAAAAAAC7E/ATe-UACI0xE/s1600/NPE1.png" /></a></div>
<br />
<br />
This is the folder where all the packages you previously downloaded are cached (from Visual Studio, NuGet Package Explorer and any other NuGet based tools). Simply copy the path and paste it in the Package Source field like this<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-UYC1fAyMOw8/U1q5si-ySgI/AAAAAAAAC7Q/68sz0t2uldc/s1600/NPE3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-UYC1fAyMOw8/U1q5si-ySgI/AAAAAAAAC7Q/68sz0t2uldc/s1600/NPE3.png" /></a></div>
<br />
<br />
This way I can quickly get to a package I just installed in my solution.<br />
<br />
NuGet Package Explorer is a powerful tool I use a lot to understand how NuGet packages are made.<br />
<br />
I hope it will help you too.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com1tag:blogger.com,1999:blog-5724958397009584147.post-12143941479459799162014-04-18T16:01:00.000-04:002014-04-18T16:01:11.194-04:00Packaging and distributing tools using NuGetNuGet is an amazing tool to manage dependencies for external libraries. As part of Visual Studio it gives us an easy way to install and update those libraries over time. Now even Microsoft use it <a href="http://blogs.msdn.com/b/dotnet/archive/2013/10/16/nuget-is-a-net-framework-release-vehicle.aspx">extensively to release updates to us</a>. But the power of NuGet doesn't stop here. Some people found imaginative ways to use NuGet like the <a href="http://scriptcs.net/" target="_blank">ScriptCs</a> project and <a href="http://chocolatey.org/" target="_blank">Chocolatey</a>. <br />
<br />
Another usage is to package and distribute utilities via NuGet packages like <a href="http://www.nuget.org/packages/FAKE/" target="_blank">FAKE</a> and <a href="http://www.nuget.org/packages/xunit.runners/" target="_blank">xUnit.Runners</a>. To create your own tool package you need to author a <a href="http://docs.nuget.org/docs/reference/nuspec-reference" target="_blank">NuSpec</a> file like this:<br />
<br />
<pre class="brush: xml"><package>
…
<files>
<b><file src="tooling\app.exe" target="tools\" /></b>
</files>
</package>
</pre><br />
The key here is to set all your file's target to <b>tools\</b>. Doing that the package will be considered a <a href="http://docs.nuget.org/docs/start-here/nuget-faq#Working_with_Packages" target="_blank">solution-level NuGet package</a> and will be available solution wide instead of only for one project.<br />
<br />
For example if I install the <a href="http://www.nuget.org/packages/xunit.runners/" target="_blank">xUnit.Runners</a> package to my solution like this:<br />
<br />
<pre>PM> Install-Package xunit.runners</pre><br />
You will see that only a new <b>package.config</b> file will be created in the <b>.nuget</b> folder at the root of your solution. This is where all solution-level packages will be referenced. Nothing will actually change inside your projects.<br />
<br />
After that all the files required to run the xUnit runner from the command line, PowerShell or a build script will be available from the <b>\packages\xunit.runners.1.9.2\tools folder</b>.<br />
<br />
The power of NuGet doesn't stop there but we'll check that in another blog post.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-82185269078270293352013-12-29T22:24:00.000-05:002013-12-29T22:24:13.021-05:00Implementing a Treemap in C#I was wondering how to implement a treemap in C#. A <a href="http://en.wikipedia.org/wiki/Treemapping">treemap</a> is a data visualisation technique that looks like this<br />
<br />
<a href="http://4.bp.blogspot.com/-Yvq0aryXoxw/UsDbaFpC5hI/AAAAAAAAC5I/qN0Cv_n6Q24/s1600/Gradient_grouped_treemap.jpg" imageanchor="1"><img border="0" src="http://4.bp.blogspot.com/-Yvq0aryXoxw/UsDbaFpC5hI/AAAAAAAAC5I/qN0Cv_n6Q24/s1600/Gradient_grouped_treemap.jpg" /></a><br />
<br />
I know a few examples exist online but between a very abstract paper like this <a href="http://www.win.tue.nl/~vanwijk/stm.pdf">one</a> and a <a href="https://github.com/imranghory/treemap-squared/blob/master/treemap-squarify.js">javascript implementation on GitHub</a> I was wondering does one could write an algorithm from scratch using a naive approach.<br />
<br />
Let's start with a simple example, the geometric series 1/2 + 1/4 + 1/8 + 1/16 + 1/32 + 1/64… would give us something like this<br />
<br />
<a href="http://3.bp.blogspot.com/-f8SFUYKyEaY/UsDbfjnnKwI/AAAAAAAAC5Q/lQK_ju5dGSY/s1600/geometric+series.png" imageanchor="1"><img border="0" src="http://3.bp.blogspot.com/-f8SFUYKyEaY/UsDbfjnnKwI/AAAAAAAAC5Q/lQK_ju5dGSY/s1600/geometric+series.png" /></a><br />
<br />
My guess would be to use a recursive algorithm to create our treemap. First we need to order all the elements from biggest to smallest. Then we start to divide the area for each elements starting with the first element which takes half the available space. After that we divide the remaining area with the rest of the elements. We repeat this process until we have all the pieces.<br />
<br />
Now if we take another more realistic example<br />
<br />
<a href="http://4.bp.blogspot.com/-yy-un-zaw0I/UsDblFL3FMI/AAAAAAAAC5Y/-ZXB_pbKxFw/s1600/treemap.png" imageanchor="1"><img border="0" src="http://4.bp.blogspot.com/-yy-un-zaw0I/UsDblFL3FMI/AAAAAAAAC5Y/-ZXB_pbKxFw/s1600/treemap.png" /></a><br />
<br />
Here every time we need to slice the area we need to make sure it won't be too small or won't look so great. If the element represent 50% or more of the total it won't be a problem but what about only 10% or 4%? I think we should set a minimum threshold for our slice, let say 25% for now. We will experiment with this value once we finish our algorithm.<br />
<br />
So what happen if the largest value is below our 25% threshold? I think we should include more elements in the slice until we reach at least 25%. For example if we have 14%, 8% and 5% the total is 27%, so our first slice will be 27% of the available space. Then we need to distribute the 3 elements in that slice. This is in essence a subset of our original problem so we can repeat the process just for that slice.<br />
<br />
By the way what orientation the first slice should be? I think we should always slice on the longest side of the area rectangle. If we have a square then it doesn't matter really.<br />
<br />
Next, how do we divide the first slice? We have 3 elements: 14%, 8% and 5%. If we look for a solution from the start we now have to following proportions: 51.8%, 29.6% and 18.5%. We can take a new slice only for the first element. And we repeat the process for the last 2 elements. Which is now 61.5% and 38.5%. At each step we need to evaluate if the slice will be horizontal or vertical depending on the the shape of the area we have.<br />
<br />
Finally, all we have to do is repeat this until all the elements are placed!<br />
<br />
Here is my implementation in LinqPad in 3 parts<br />
<br />
<b>Slice calculation</b><br />
<pre class="brush: csharp">public Slice<T> GetSlice<T>(IEnumerable<Element<T>> elements, double totalSize,
double sliceWidth)
{
if (!elements.Any()) return null;
if (elements.Count() == 1) return new Slice<T>
{ Elements = elements, Size = totalSize };
var sliceResult = GetElementsForSlice(elements, sliceWidth);
return new Slice<T>
{
Elements = elements,
Size = totalSize,
SubSlices = new[]
{
GetSlice(sliceResult.Elements, sliceResult.ElementsSize, sliceWidth),
GetSlice(sliceResult.RemainingElements, 1 - sliceResult.ElementsSize,
sliceWidth)
}
};
}
private SliceResult<T> GetElementsForSlice<T>(IEnumerable<Element<T>> elements,
double sliceWidth)
{
var elementsInSlice = new List<Element<T>>();
var remainingElements = new List<Element<T>>();
double current = 0;
double total = elements.Sum(x => x.Value);
foreach (var element in elements)
{
if (current > sliceWidth)
remainingElements.Add(element);
else
{
elementsInSlice.Add(element);
current += element.Value / total;
}
}
return new SliceResult<T>
{
Elements = elementsInSlice,
ElementsSize = current,
RemainingElements = remainingElements
};
}
public class SliceResult<T>
{
public IEnumerable<Element<T>> Elements { get; set; }
public double ElementsSize { get; set; }
public IEnumerable<Element<T>> RemainingElements { get; set; }
}
public class Slice<T>
{
public double Size { get; set; }
public IEnumerable<Element<T>> Elements { get; set; }
public IEnumerable<Slice<T>> SubSlices { get; set; }
}
public class Element<T>
{
public T Object { get; set; }
public double Value { get; set; }
}
</pre>
<br />
<b>Generating rectangles using leaf slice (slice with only one element in it)</b><br />
<pre class="brush: csharp">public IEnumerable<SliceRectangle<T>> GetRectangles<T>(Slice<T> slice, int width,
int height)
{
var area = new SliceRectangle<T>
{ Slice = slice, Width = width, Height = height };
foreach (var rect in GetRectangles(area))
{
// Make sure no rectangle go outside the original area
if (rect.X + rect.Width > area.Width) rect.Width = area.Width - rect.X;
if (rect.Y + rect.Height > area.Height) rect.Height = area.Height - rect.Y;
yield return rect;
}
}
private IEnumerable<SliceRectangle<T>> GetRectangles<T>(
SliceRectangle<T> sliceRectangle)
{
var isHorizontalSplit = sliceRectangle.Width >= sliceRectangle.Height;
var currentPos = 0;
foreach (var subSlice in sliceRectangle.Slice.SubSlices)
{
var subRect = new SliceRectangle<T> { Slice = subSlice };
int rectSize;
if (isHorizontalSplit)
{
rectSize = (int)Math.Round(sliceRectangle.Width * subSlice.Size);
subRect.X = sliceRectangle.X + currentPos;
subRect.Y = sliceRectangle.Y;
subRect.Width = rectSize;
subRect.Height = sliceRectangle.Height;
}
else
{
rectSize = (int)Math.Round(sliceRectangle.Height * subSlice.Size);
subRect.X = sliceRectangle.X;
subRect.Y = sliceRectangle.Y + currentPos;
subRect.Width = sliceRectangle.Width;
subRect.Height = rectSize;
}
currentPos += rectSize;
if (subSlice.Elements.Count() > 1)
{
foreach (var sr in GetRectangles(subRect))
yield return sr;
}
else if (subSlice.Elements.Count() == 1)
yield return subRect;
}
}
public class SliceRectangle<T>
{
public Slice<T> Slice { get; set; }
public int X { get; set; }
public int Y { get; set; }
public int Width { get; set; }
public int Height { get; set; }
}
</pre>
<br />
<b>Drawing the rectangles in WinForm</b><br />
<pre class="brush: csharp">public void DrawTreemap<T>(IEnumerable<SliceRectangle<T>> rectangles, int width,
int height)
{
var font = new Font("Arial", 8 );
var bmp = new Bitmap(width, height);
var gfx = Graphics.FromImage(bmp);
gfx.FillRectangle(Brushes.Blue, new RectangleF(0, 0, width, height));
foreach (var r in rectangles)
{
gfx.DrawRectangle(Pens.Black,
new Rectangle(r.X, r.Y, r.Width - 1, r.Height - 1));
gfx.DrawString(r.Slice.Elements.First().Object.ToString(), font,
Brushes.White, r.X, r.Y);
}
var form = new Form() { AutoSize = true };
form.Controls.Add(new PictureBox()
{ Width = width, Height = height, Image = bmp });
form.ShowDialog();
}
</pre>
<br />
<b>And finally to generate a Treemap in LinqPad</b><br />
<pre class="brush: csharp">void Main()
{
const int Width = 400;
const int Height = 300;
const double MinSliceRatio = 0.35;
var elements = new[] { 24, 45, 32, 87, 34, 58, 10, 4, 5, 9, 52, 34 }
.Select (x => new Element<string> { Object = x.ToString(), Value = x })
.OrderByDescending (x => x.Value)
.ToList();
var slice = GetSlice(elements, 1, MinSliceRatio).Dump("Slices");
var rectangles = GetRectangles(slice, Width, Height)
.ToList().Dump("Rectangles");
DrawTreemap(rectangles, Width, Height);
}
</pre>
<br />
<b>References:</b><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://en.wikipedia.org/wiki/Treemapping"><span lang="en-CA">http://en.wikipedia.org/wiki/Treemapping</span></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://www.win.tue.nl/~vanwijk/stm.pdf"><span lang="en-CA">http://www.win.tue.nl/~vanwijk/stm.pdf</span></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://stackoverflow.com/questions/3438252/creating-a-treemap-visualization"><span lang="en-CA">http://stackoverflow.com/questions/3438252/creating-a-treemap-visualization</span></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="https://github.com/imranghory/treemap-squared/blob/master/treemap-squarify.js"><span lang="en-CA">https://github.com/imranghory/treemap-squared/blob/master/treemap-squarify.js</span></a></div>
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2tag:blogger.com,1999:blog-5724958397009584147.post-75486335904201131532013-08-28T22:32:00.000-04:002013-08-28T22:32:31.070-04:00Windows Azure Caching and transient faultsWhen using remote services over the wire we should always plan for transient failures. <a href="http://www.windowsazure.com/en-us/services/caching/">Windows Azure Caching</a> like any services in an Azure world is prone to such problem. Out of the box making calls to the cache server will fail from time to time due to network issues. Typically you will get those kind of exceptions:<br />
<pre>Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later.</pre><pre>Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out.</pre><pre>Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server.</pre><br />
For that reason it is a best practice to implement some kind of retry logic around your code calling the cache server. We could have used the <a href="http://msdn.microsoft.com/en-us/library/hh680934(v=pandp.50).aspx">Transient Application Block</a> to manage that. But a few months ago I found somewhere that from time to time the DataCache object lose it's internal connection to the cache server. A simple way to fix this is to re-create a DataCache instance and retry the operation.<br />
<br />
In the implementation below I'm keeping a reference to the DataCacheFactory and DataCache objects (another best practice). The CreateDataCache factory method will come handy later.<br />
<br />
<pre class="brush: csharp">public class CachingService
{
private DataCacheFactory cacheFactory;
private DataCache cache;
private DataCache Cache
{
get
{
if (this.cache == null)
{
this.CreateDataCache();
}
return this.cache;
}
}
private void CreateDataCache()
{
this.cacheFactory = new DataCacheFactory();
this.cache = this.cacheFactory.GetDefaultCache();
}
// ...
}
</pre><br />
Then I have this SafeCallFunction I use whenever I want to work with the DataCache object. Notice that the only thing I do to retry the operation is to call the factory method to re-create the DataCache object.<br />
<br />
<pre class="brush: csharp">private object SafeCallFunction(Func<object> function)
{
try
{
return function.Invoke();
}
catch (DataCacheException)
{
// Retry by first re-creating the DataCache
try
{
this.CreateDataCache();
return function.Invoke();
}
catch (DataCacheException)
{
// Log error
}
}
return null;
}
</pre><br />
Finally in the rest of the class I can use the SafeCallFunction like this <br />
<pre class="brush: csharp">public object CacheGet(string key)
{
return this.SafeCallFunction(() => this.Cache.Get(key));
}
public void CachePut(string key, object cacheObject)
{
this.SafeCallFunction(() => this.Cache.Put(key, cacheObject));
}
public void CacheRemove(string key)
{
this.SafeCallFunction(() => this.Cache.Remove(key));
}
</pre><br />
So far after a few weeks of using this implementation the single retry never failed on us. Before that we had around 5-10 failures daily for about 500k calls to the cache server. I would still recommend using a more robust retry policy with Windows Azure Caching but I think it's interesting to know that simply instantiating a new DataCache can fix most failures.<br />
<br />
<h3>References</h3><a href="http://msdn.microsoft.com/en-us/library/gg278356.aspx">Caching in Windows Azure</a> <br />
<a href="http://blogs.msdn.com/b/jagan_peri/archive/2012/09/09/best-practices-for-using-windows-azure-cache-windows-server-appfabric-cache.aspx">Best Practices for using Windows Azure Cache</a> <br />
<a href="http://msdn.microsoft.com/en-us/library/hh916611.aspx#section_8">Optimization Guidance for Windows Azure Caching</a> <br />
<a href="http://msdn.microsoft.com/en-us/library/hh680934(v=pandp.50).aspx">The Transient Fault Handling Application Block</a>Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com1tag:blogger.com,1999:blog-5724958397009584147.post-78227823449247767692013-07-31T22:13:00.000-04:002013-07-31T22:19:43.717-04:00Using Windows Azure Caching efficiently across multiple Cloud Service roles <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/cache" target="_blank">Windows Azure Caching</a> is a great way to improve performance of your Azure application at no additional cost. The cache is running alongside your application in Cloud Service roles. The only thing you need to decide is how much memory of the role you want use for it (for co-located cache role). You can also dedicate the all memory of a role to caching if you want (with dedicated cache role). Roles that host caching are called cache clusters.<br />
<br />
Starting with Azure Caching is so easy that it can be a while before you fully understand the best way to use it. On a recent project my first tough was to enable caching on all the Cloud Service roles as co-located service. This was causing us problems.<br />
<br />
First of all, been a developer I debug my application using the local Azure compute emulator. The emulator runs one cache service for each instances of roles with cache clusters. The application has 2 web roles and 1 worker role so when I start a debugging session with multiple instances per role I need a lot of memory to run everything. More importantly, cache clusters do not share cached data between each other. This caused us to have stale data in the application.<br />
<br />
That is when I figured out that I needed to read a bit more on Azure Caching if I was to use it efficiently.<br />
<br />
<h1>Understanding cache clusters</h1><br />
When you enable caching on an Azure role each instance of that role will run a cache service using a portion of the memory (or all of it if it's a dedicated cache role). The cache services running on each instance of a single role are managed as a single cache cluster. Cache services can talk to each other and synchronize data but only inside the same cache cluster (same role). That is why enabling caching of many roles might not be the best thing to do.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-p2lRfWtJWMU/Ufce8mCmdHI/AAAAAAAAC3c/aZP_fPQQgIs/s1600/clusters+not+connecting.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-p2lRfWtJWMU/Ufce8mCmdHI/AAAAAAAAC3c/aZP_fPQQgIs/s1600/clusters+not+connecting.png" /></a></div><br />
Another thing to mention is that cache clusters can only be created on small role instances or bigger. The reason is that with <a href="https://www.windowsazure.com/en-us/pricing/details/cloud-services/" target="_blank">extra small instance you only get 768MB of RAM</a> which is pretty much all used up by anything you run on those instances.<br />
<br />
Now the enable a cache cluster on your role go to the role property page on the Caching tab. <br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-KlXKhQ4pm8Y/UfciWMp2orI/AAAAAAAAC30/89AugbgpXVk/s1600/caching+tab.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-KlXKhQ4pm8Y/UfciWMp2orI/AAAAAAAAC30/89AugbgpXVk/s1600/caching+tab.png" /></a></div><br />
Here you will notice that I also enabled notifications which will allow us to efficiently use local caches later.<br />
<br />
For more information on the different configuration options for cache clusters go <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/cache/#enable-caching">here</a>.<br />
<br />
<h1>Configuring roles to use cache clients</h1><br />
Now that we took care of the server side of caching configuration let's talk about the client side. Each instances of each roles inside the same cloud deployment can connect to a cache cluster. If you run only one cluster then you are guarantied to access the same cached data from whatever role you are inside your application (as long you have a valid configuration).<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-HS8og8fUC2s/UfceVjxlq_I/AAAAAAAAC3U/0Ud2nIe1uAY/s1600/cluster+and+clients.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-HS8og8fUC2s/UfceVjxlq_I/AAAAAAAAC3U/0Ud2nIe1uAY/s1600/cluster+and+clients.png" /></a></div><br />
One nice feature we can enable in each role configuration is the local cache client. With this we can cache data locally in a role instance memory the data we recently fetched from the cache cluster for even faster access. Remember the Notification option we enabled on the server side? Using the configuration below in the Web.config or App.config of your role will ensure data stored in the local cache client gets updated whenever the cache server version of that data changes. Basically, the local cache client will invalidate data based on notifications received from the cache cluster.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-GgYQkUWTYdw/Ufcj6j-FwUI/AAAAAAAAC4E/tzoL4v0wIt8/s1600/cache+client+config.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-GgYQkUWTYdw/Ufcj6j-FwUI/AAAAAAAAC4E/tzoL4v0wIt8/s1600/cache+client+config.png" /></a></div><br />
For more information on client side configuration go <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/cache/#NuGet">here</a>.<br />
<br />
<h1>Other concerns</h1><br />
This post is only about an overview of the consideration of running multiple clusters versus a single one. Using Azure Caching there are a lot more configuration options you need to take a look at <a href="http://msdn.microsoft.com/en-us/library/windowsazure/hh697523.aspx">here</a>. Also really important is <a href="http://msdn.microsoft.com/en-us/library/windowsazure/hh914165.aspx">how to use Azure Caching</a> in your application.<br />
<br />
<h1>Conclusions</h1><br />
I've spend a lot of time figuring out how all of this was working. I hope this post will help you with your learning experience.<br />
<br />
<h3>Other useful links</h3><br />
<ul><li><a href="http://msdn.microsoft.com/en-us/library/windowsazure/gg278356.aspx">Caching in Windows Azure</a></li>
<li><a href="http://msdn.microsoft.com/en-us/library/windowsazure/hh916611.aspx">Optimization Guidance for Windows Azure Caching</a></li>
<li><a href="http://blogs.msdn.com/b/jagan_peri/archive/2012/09/09/best-practices-for-using-windows-azure-cache-windows-server-appfabric-cache.aspx">Best Practices for using Windows Azure Cache</a></li>
</ul><br />
<br />
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-13688670170825456102013-07-22T22:33:00.000-04:002013-07-22T22:33:54.232-04:00Handling Azure Storage Queue poison messagesThis post will talk about what to do now that we are <a href="http://pascallaurin42.blogspot.com/2013/06/windows-azure-storage-queue-with-error.html">handling poison messages in our Azure Storage Queues</a>.<br />
<br />
First, let's review what we'll done so far. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-fppsPrp7Dro/UcRiSiq4aLI/AAAAAAAAC14/-kOex1MLYIY/s1600/chart2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-fppsPrp7Dro/UcRiSiq4aLI/AAAAAAAAC14/-kOex1MLYIY/s1600/chart2.png" /></a></div>
<br />
Messages that continuously fail to process will end up in the Error Queue. Someone asked me why do we need error queues at all? We could simply log the errors and delete the message right? Well, if you have a really efficient and pro-active <a href="http://en.wikipedia.org/wiki/DevOps">DevOps</a> team I suppose logging errors along with the original messages ought to be enough. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-et2WHZpIrt0/Ueybw2M3E6I/AAAAAAAAC2M/Ug6PcYsnUtU/s1600/Error+Queues+with+Logs+-+New+Page.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-et2WHZpIrt0/Ueybw2M3E6I/AAAAAAAAC2M/Ug6PcYsnUtU/s1600/Error+Queues+with+Logs+-+New+Page.png" /></a></div>
Someone will review why the message failed and if it was only a transient error then he could send the original message again in the queue.<br />
<br />
We could also store failed messages into an <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/" target="_blank">Azure Storage Table</a>. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-hWRQpbkcJ64/UeydlcsnQ5I/AAAAAAAAC2c/JWLEOIk0H5k/s1600/Error+Queues+with+Table+-+New+Page.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-hWRQpbkcJ64/UeydlcsnQ5I/AAAAAAAAC2c/JWLEOIk0H5k/s1600/Error+Queues+with+Table+-+New+Page.png" /></a></div>
Then we could simply monitor new entries in this table and act on it. Again, the original message should be stored in the table so we could send it again if we choose.<br />
<br />
I think the best reason to use a queue for error message is if you want to have an administrative tools to monitor, review and re-send messages. In this case the <i>queue</i> mechanics let's you handle those messages like any other process using queues do.<br />
<br />
For me one of the unpleasant side effect of using error queues is that all the queues in my system are now multiplied by two (one normal and one error queue). It's not too bad if your naming scheme is consistent but even then if you do operational work using a tool like <a href="http://www.cerebrata.com/products/azure-management-studio/introduction" target="_blank">Cerebrata Azure Management Studio</a> or even from Visual Studio's Server Explorer you will feel overwhelmed by the quantity of queues.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-nyh52s1gy1s/UeygaA9oqoI/AAAAAAAAC2s/81f7xWoUWvw/s1600/Cerebrate+Azure+Management+Studio.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-nyh52s1gy1s/UeygaA9oqoI/AAAAAAAAC2s/81f7xWoUWvw/s1600/Cerebrate+Azure+Management+Studio.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Managing queue messages with Cerebrata Azure Management Studio</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-T1O6tHFwEyE/UeyjnYIaJFI/AAAAAAAAC28/Zsohvt6p7v4/s1600/Visual+Studio+Server+Explorer.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-T1O6tHFwEyE/UeyjnYIaJFI/AAAAAAAAC28/Zsohvt6p7v4/s1600/Visual+Studio+Server+Explorer.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Managing queue messages with Visual Studio Server Explorer</td></tr>
</tbody></table>
<br />
Whatever you do, I suggest you always at least log failures properly. Later while debugging the issue you will be thankful to easily match the failure logs with the message who caused it.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2tag:blogger.com,1999:blog-5724958397009584147.post-18240099478992193342013-06-21T10:36:00.000-04:002013-07-23T12:48:27.919-04:00Windows Azure Storage Queue with error queues<a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/queue-service/">Windows Azure Storage Queue</a> are an excellent way to execute tasks asynchronously in Azure. For example a web application could let all the heavy processing to a Worker Role instead of doing it itself. That way requests will complete faster. Queues can also be used to decouple communication between two separated applications or two components of the same application.<br />
<br />
One problem with asynchronous processing is what to do when the operation fails? When using synchronous patterns like a direct call we usually return an error code or a message or throw an exception. When we use queues to delegate the execution to another process we can't notify the originator directly. One thing we can do is to send a message back to the originator through another queue, like a callback. This is interesting when the process produce a result normally, an error in this case is only another kind of result.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-NqcJ1unExTs/UcG4y1dgjhI/AAAAAAAAC1Y/dNnx_xz8A7g/s1600/chart1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="118" src="http://4.bp.blogspot.com/-NqcJ1unExTs/UcG4y1dgjhI/AAAAAAAAC1Y/dNnx_xz8A7g/s320/chart1.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Another way is to create an <i>error</i> queue (or <i>dead letter</i> queue) for poison messages where we put all messages that failed processing. This way we get a list of all the failing messages to review, find out what was the problem with them and figure out what to do about it. For example we can retry the message by moving it back to the main queue so it can be processed again.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-fppsPrp7Dro/UcRiSiq4aLI/AAAAAAAAC10/2CxQzwgjHxM/s1600/chart2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="118" src="http://3.bp.blogspot.com/-fppsPrp7Dro/UcRiSiq4aLI/AAAAAAAAC10/2CxQzwgjHxM/s320/chart2.png" width="320" /></a></div>
<br />
Now, let see how we can implement an error queue using Windows Azure Storage Queue.<br />
<br />
<h3>
Implementation of an error queue</h3>
<br />
First we will initialize the queues. For each queue we also create an '<queuename>-error' queue.<br />
<pre class="brush: csharp">var storageAccount = CloudStorageAccount.Parse("UseDevelopmentStorage=true");
var queueClient = storageAccount.CreateCloudQueueClient();
this.taskQueueReference = queueClient.GetQueueReference("task");
this.taskErrorQueueReference = queueClient.GetQueueReference("task-error");
this.taskQueueReference.CreateIfNotExists();
this.taskErrorQueueReference.CreateIfNotExists();
</pre>
<br />
Next we add a few messages with one that will cause the processing to fail (to simulate failures)<br />
<pre class="brush: csharp">this.taskQueueReference.AddMessage(
new CloudQueueMessage("Message " + DateTime.UtcNow.Ticks));
this.taskQueueReference.AddMessage(
new CloudQueueMessage("Message " + DateTime.UtcNow.Ticks));
this.taskQueueReference.AddMessage(
new CloudQueueMessage("Error " + DateTime.UtcNow.Ticks));
</pre>
<br />
Finally the code to actually poll the queue for messages. Usually polling is done in an infinite loop but when no message is fetched it is a good practices to wait a while before polling again to prevent unnecessary transaction cost and IO (each call to GetMessages is 1 transaction). Depending on the need for the queue to react rapidly to new messages this may go between a few seconds for critical tasks to a few minutes for non critical tasks. Also I'm using a retry mechanism here, meaning that I'll try to process a message a few times before I really consider it in error (poison). If we don't delete a message after fetching it then after some time it goes back in the queue to be processed again. This mean all tasks we want to process using queues should be <a href="http://stackoverflow.com/questions/1077412/what-is-an-idempotent-operation">idempotent</a><br />
<pre class="brush: csharp">private void PollQueue()
{
IEnumerable<CloudQueueMessage> messages;
do
{
messages = this.taskQueueReference
.GetMessages(8, visibilityTimeout: TimeSpan.FromSeconds(10));
foreach (var message in messages)
{
bool result = false;
try
{
result = this.ProcessMessage(message);
if (result) this.taskQueueReference.DeleteMessage(message);
}
catch (Exception ex)
{
this.Log(message.AsString, ex);
}
if (!result && message.DequeueCount >= 3)
{
this.taskErrorQueueReference.AddMessage(message);
this.taskQueueReference.DeleteMessage(message);
}
}
} while (messages.Any());
}
private bool ProcessMessage(CloudQueueMessage message)
{
if (message.AsString.StartsWith("Error")) throw new Exception("Error!");
return true;
}
</pre>
<br />
First I'm fetching messages by batch of 8 in this case. In one transaction you can fetch between 1 and 32 messages. Also I set the visibilityTimeout to 10 seconds. This means the messages won't be visible to anyone during that time. Usually you want to set the visibility timeout based on how much time should be required to process all the messages of the batch. If we don't have the time to delete the messages from the queue before the timeout elapse another worker could fetch the message and start processing it again. So we should balance the time to process all the messages in one batch with how much time we want to allow between retries.<br />
<br />
Next we process the message. If the processing is successful we simply return true so the message can be deleted from the queue. If processing failed we have two options, return false or throw an exception. I simply return false instead of throwing an exception most of the time when the failure is expected.<br />
<br />
Finally, we check how many times we unsuccessfully tried to process the message and if we reached our limit (in this case 3 times). If we did then it's time to send that message to the error queue and delete it from the normal queue.<br />
<br />
Next time we will look at how we want to handle the messages in the error queue. You can find this post <a href="http://pascallaurin42.blogspot.com/2013/07/handling-azure-storage-queue-poison.html" target="">here</a>.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com4tag:blogger.com,1999:blog-5724958397009584147.post-72545287261131180352013-05-09T22:58:00.001-04:002013-05-09T22:58:54.524-04:00Using Azure Blob Storage to store documentsLast time I wrote about <a href="http://pascallaurin42.blogspot.ca/2013/04/document-oriented-database-with-azure.html">Implementing a document oriented database with the Windows Azure Table Storage Service</a> I was using the <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/">Table Storage Service</a> to store serialized documents into an entity's property. While it is an easy way to store complex objects the table storage is usually meant to storage primitives like int, bool, date and simple strings. However there is another service in the <a href="http://www.windowsazure.com/en-us/manage/services/storage/">Windows Azure Storage</a> family who is better suited to store documents: the <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/">Blob Storage Service</a>. The blob storage use the metaphor of files which is in essence documents.<br />
<br />
Now I will adapt the Repository I did in my previous <a href="http://pascallaurin42.blogspot.ca/2013/04/document-oriented-database-with-azure.html">post</a> to use the blob storage this time. I'll only walk through the changes I'm making here.<br />
<br />
<h3>Constructor</h3><br />
First, we need to create a <a href="http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.cloudblobcontainer.aspx">CloudBlobContainer</a> instance in the constructor. Please note that for blob storage container names are required to be <b>lower-case</b>.<br />
<br />
<pre class="brush: csharp">public class ProjectRepository
{
private CloudTable table;
private CloudBlobContainer container;
public ProjectRepository()
{
var connectionString = "...";
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(connectionString);
var tableClient = storageAccount.CreateCloudTableClient();
this.table = tableClient.GetTableReference("Project");
this.table.CreateIfNotExists();
var blobClient = storageAccount.CreateCloudBlobClient();
this.container = blobClient.GetContainerReference("project");
this.container.CreateIfNotExists();
}
// ...
}
</pre><br />
<h3>Insert</h3><br />
Next for the Insert method, we no longer store the document in a property of the <a href="http://pascallaurin42.blogspot.com/2013/03/using-azure-table-storage-with-dynamic.html">ElasticTableEntity</a> object. Instead we want to serialize the document into the JSON format and upload it as a file to the blob storage and set the ContentType of that file to <i>application/json</i>. For the blob name (or path) the pattern I'm using looks like this: <b>{document-type}/{partition-key}/{row-key}</b>.<br />
<br />
<pre class="brush: csharp">public void Insert(Project project)
{
project.Id = Guid.NewGuid();
var document = JsonConvert.SerializeObject(project,
Newtonsoft.Json.Formatting.Indented);
var partitionKey = project.Owner.ToString();
var rowKey = project.Id.ToString();
UploadDocument(partitionKey, rowKey, document);
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = partitionKey;
entity.RowKey = rowKey;
entity.Name = project.Name;
entity.StartDate = project.StartDate;
entity.TotalTasks = project.Tasks.Count();
this.table.Execute(TableOperation.Insert(entity));
}
private void UploadDocument(string partitionKey, string rowKey, string document)
{
var filename = string.Format(@"project\{0}\{1}.json", partitionKey, rowKey);
var blockBlob = this.container.GetBlockBlobReference(filename);
using (var memory = new MemoryStream())
using (var writer = new StreamWriter(memory))
{
writer.Write(document);
writer.Flush();
memory.Seek(0, SeekOrigin.Begin);
blockBlob.UploadFromStream(memory);
}
blockBlob.Properties.ContentType = "application/json";
blockBlob.SetProperties();
}
</pre><br />
<h3>Load</h3><br />
For the Load method we can get the blob name using the PartitionKey and RowKey then download the document from blob storage. In DownloadDocument I'm using a MemoryStream and StreamReader to get the serialized document as a string.<br />
<br />
<pre class="brush: csharp">public Project Load(string partitionKey, string rowKey)
{
var blobName = string.Format(@"project\{0}\{1}.json", partitionKey, rowKey);
var document = this.DownloadDocument(blobName);
return JsonConvert.DeserializeObject<Project>(document);
}
private string DownloadDocument(string blobName)
{
var blockBlob = this.container.GetBlockBlobReference(blobName);
using (var memory = new MemoryStream())
using (var reader = new StreamReader(memory))
{
blockBlob.DownloadToStream(memory);
memory.Seek(0, SeekOrigin.Begin);
return reader.ReadToEnd();
}
}</pre><br />
<h3>List</h3><br />
In the first List method we want to get all documents of the same partition. We can do that by directly using the ListBlobs method of <a href="http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.blob.cloudblobdirectory.aspx">CloudBlobDirectory</a>. For the ListWithTasks method we still need to query the table storage first to know which documents contain at least one task. Then with the entities we'll know the RowKey value of those documents so we can simply call the Load method we just saw.<br />
<br />
<pre class="brush: csharp">public IEnumerable<Project> List(string partitionKey)
{
var listItems = this.container
.GetDirectoryReference("project/" + partitionKey).ListBlobs();
return listItems.OfType<CloudBlockBlob>()
.Select(x => this.DownloadDocument(x.Name))
.Select(document => JsonConvert.DeserializeObject<Project>(document));
}
public IEnumerable<Project> ListWithTasks(string partitionKey)
{
var query = new TableQuery<ElasticTableEntity>()
.Select(new [] { "RowKey" })
.Where(TableQuery.CombineFilters(
TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, partitionKey),
TableOperators.And,
TableQuery.GenerateFilterConditionForInt("TotalTasks",
QueryComparisons.GreaterThan, 0)));
dynamic entities = table.ExecuteQuery(query).ToList();
foreach (var entity in entities)
yield return this.Load(partitionKey, entity.RowKey);
}
</pre><br />
<h3>Update</h3><br />
To update a document now we also need to serialize and upload the new version to blob storage.<br />
<br />
<pre class="brush: csharp">public void Update(Project project)
{
var document = JsonConvert.SerializeObject(project,
Newtonsoft.Json.Formatting.Indented);
var partitionKey = project.Owner.ToString();
var rowKey = project.Id.ToString();
UploadDocument(partitionKey, rowKey, document);
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = partitionKey;
entity.RowKey = rowKey;
entity.ETag = "*";
entity.Name = project.Name;
entity.StartDate = project.StartDate;
entity.TotalTasks = project.Tasks != null ? project.Tasks.Count() : 0;
this.table.Execute(TableOperation.Replace(entity));
}
</pre><br />
<h3>Delete</h3><br />
Finally, deleting a document now requires us to call Delete on the CloudBlobContainer reference.<br />
<br />
<pre class="brush: csharp">public void Delete(Project project)
{
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = project.Owner.ToString();
entity.RowKey = project.Id.ToString();
entity.ETag = "*";
this.table.Execute(TableOperation.Delete(entity));
this.DeleteDocument(entity.PartitionKey, entity.RowKey);
}
public void Delete(string partitionKey, string rowKey)
{
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = partitionKey;
entity.RowKey = rowKey;
entity.ETag = "*";
this.table.Execute(TableOperation.Delete(entity));
this.DeleteDocument(partitionKey, rowKey);
}
private void DeleteDocument(string partitionKey, string rowKey)
{
var blobName = string.Format(@"project\{0}\{1}.json", partitionKey, rowKey);
var blockBlob = this.container.GetBlockBlobReference(blobName);
blockBlob.Delete(DeleteSnapshotsOption.IncludeSnapshots);
}
</pre><br />
<h3>Conclusion</h3><br />
Using both Tables and Blobs Storage Services we can get the best of both worlds. We can query for document's properties with table storage and we can store documents larger than 64KB in blob storage. Of course now almost all operations on my Repository requires two calls to Azure. Currently those are done sequentially, waiting for the first call to complete before the doing the second call. I should fix that by using the asynchronous variants of storage service methods like the BeginDelete/EndDelete method pair on CloudBlobContainer.<br />
<br />
I hope this post is giving you ideas on new and clever ways you can use the Windows Azure Storage Services in your projects.<br />
<br />
<h3>See also</h3>- <a href="http://pascallaurin42.blogspot.com/2013/03/using-azure-table-storage-with-dynamic.html">Using Azure Table Storage with dynamic table entities</a><br />
- <a href="http://pascallaurin42.blogspot.ca/2013/04/document-oriented-database-with-azure.html">Document oriented database with Azure Table Storage Service</a> Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2tag:blogger.com,1999:blog-5724958397009584147.post-39383524489493431722013-04-09T22:10:00.000-04:002013-05-09T23:01:27.198-04:00Document oriented database with Azure Table Storage Service<h3>What is a Document database?</h3><br />
A document store or <a href="http://en.wikipedia.org/wiki/Document-oriented_database">document oriented database</a> like <a href="http://ravendb.net/">RavenDB</a> is a kind of <a href="http://en.wikipedia.org/wiki/NoSQL">NoSQL</a> database where we store semi structured data in documents and use a key to retrieve existing documents.<br />
<br />
The difference with a relational database is that a complex data structure needs to be represented by entities and relations in a <a href="http://en.wikipedia.org/wiki/Relational_database_management_system">RDMS</a> which means using many tables, joints and constraints. In a document database the whole graph of entities is stored as a single document. Of course we still need to handle some relations between documents or graphs of entities. This is done by storing other documents key inside the document and acting like a foreign keys.<br />
<br />
Another way to see documents is to think that all tables related together with a <a href="http://en.wikipedia.org/wiki/Foreign_key#CASCADE"><i>delete cascade constraints</i></a> on the relations are part of the same document. If a piece of data from a table can only exists if related data from another table also exists it means both should be part of the same document.<br />
<br />
The concept of document relates well with Aggregates of <a href="http://www.domaindrivendesign.org/">Domain Driven Design</a>.<br />
<br />
<h3>Implementing a document database with the Azure Table Storage Service</h3><br />
Looking at RavenDB I was wondering if it was possible to use similar patterns but with the <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/">Windows Azure Table Storage Service</a> instead of the file system as RavenDB is using.<br />
<br />
A good storage format for our documents is the JSON representation. Using a serialization library like Json.Net it will be easy to convert our data in JSON.<br />
<br />
Using the Table Storage Service also requires us to provide a PartionKey for our document, in a typical multi-tenant database we could use this to <i>'partition'</i> the data per tenant. <br />
<br />
A document in a document store is easy to retrieve when we know the key to fetch it directly, but sometime we don't have that information. We might also want to query documents using a filter expression. In a relational database filtering in a query is easy but in a document store it requires a bit more efforts. RavenDB let us define indexes we could use to filter documents in queries. With the Table Store Service we can use additional properties to store information we want to filter on, acting like the indexes.<br />
<br />
In order to create a very light weight Document Store with the Table Storage Service I will use my <a href="http://pascallaurin42.blogspot.ca/2013/03/using-azure-table-storage-with-dynamic.html">ElasticTableEntity</a> class from a previous post.<br />
<br />
First let me show you my domain entities for this demo. A simple Project class which may have many Tasks associated to it.<br />
<br />
<pre class="brush: csharp">public class Project
{
public Guid Owner { get; set; }
public Guid Id { get; set; }
public string Name { get; set; }
public DateTime StartDate { get; set; }
public int Status { get; set; }
public List<Task> Tasks { get; set; }
}
public class Task
{
public string Name { get; set; }
public bool IsCompleted { get; set; }
}
</pre><br />
Now let's take a look a typical <a href="http://martinfowler.com/eaaCatalog/repository.html">Repository</a> implementation for the Projects. You will need both <a href="http://nuget.org/packages/WindowsAzure.Storage/">WindowsAzure.Storage</a> and <a href="http://nuget.org/packages/Newtonsoft.Json/">Newtonsoft.Json</a> packages from NuGet for this part.<br />
<br />
<pre class="brush: csharp">public class ProjectRepository
{
private CloudTable table;
public ProjectRepository()
{
var connectionString = "...";
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(connectionString);
var client = storageAccount.CreateCloudTableClient();
this.table = client.GetTableReference("Project");
this.table.CreateIfNotExists();
}
public void Insert(Project project)
{
project.Id = Guid.NewGuid();
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = project.Owner.ToString();
entity.RowKey = project.Id.ToString();
entity.Document = JsonConvert.SerializeObject(project,
Newtonsoft.Json.Formatting.Indented);
// Additional fields for querying (indexes)
entity.Name = project.Name;
entity.StartDate = project.StartDate;
entity.TotalTasks = project.Tasks.Count();
this.table.Execute(TableOperation.Insert(entity));
}
public IEnumerable<Project> List(string partitionKey)
{
var query = new TableQuery<ElasticTableEntity>()
.Select(new [] { "Document" })
.Where(TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, partitionKey));
dynamic entities = table.ExecuteQuery(query).ToList();
foreach (var entity in entities)
{
var document = (string)entity.Document.StringValue;
yield return JsonConvert.DeserializeObject<Project>(document);
}
}
public IEnumerable<Project> ListWithTasks(string partitionKey)
{
var query = new TableQuery<ElasticTableEntity>()
.Select(new [] { "Document" })
.Where(TableQuery.CombineFilters(
TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, partitionKey),
TableOperators.And,
TableQuery.GenerateFilterConditionForInt("TotalTasks",
QueryComparisons.GreaterThan, 0)));
dynamic entities = table.ExecuteQuery(query).ToList();
foreach (var entity in entities)
{
var document = (string)entity.Document.StringValue;
yield return JsonConvert.DeserializeObject<Project>(document);
}
}
public Project Load(string partitionKey, string rowKey)
{
var query = new TableQuery<ElasticTableEntity>()
.Select(new [] { "Document" })
.Where(TableQuery.CombineFilters(
TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, partitionKey),
TableOperators.And,
TableQuery.GenerateFilterCondition("RowKey",
QueryComparisons.Equal, rowKey)));
dynamic entity = table.ExecuteQuery(query).SingleOrDefault();
if (entity != null)
{
var document = (string)entity.Document.StringValue;
return JsonConvert.DeserializeObject<Project>(document);
}
return null;
}
public void Update(Project project)
{
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = project.Owner.ToString();
entity.RowKey = project.Id.ToString();
entity.ETag = "*";
entity.Document = JsonConvert.SerializeObject(project,
Newtonsoft.Json.Formatting.Indented);
// Additional fields for querying (indexes)
entity.Name = project.Name;
entity.StartDate = project.StartDate;
entity.TotalTasks = project.Tasks.Count();
this.table.Execute(TableOperation.Replace(entity));
}
public void Delete(Project project)
{
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = project.Owner.ToString();
entity.RowKey = project.Id.ToString();
entity.ETag = "*";
this.table.Execute(TableOperation.Delete(entity));
}
public void Delete(string partitionKey, string rowKey)
{
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = partitionKey;
entity.RowKey = rowKey;
entity.ETag = "*";
this.table.Execute(TableOperation.Delete(entity));
}
}
</pre><br />
We could refactor the code to reduce duplication but the point was to show you how to dynamically create a <i>Document</i> property to store the actual serialized document and how to handle the basic CRUD operations. You can also see how to dynamically add other properties to store extra information on the document. This is useful for the <i>LoadWithTasks</i> method which fetch only Projects with at least one Task on it.<br />
<br />
Finally let's take a look at a few examples on how to use the ProjectRepository itself (in <a href="http://www.linqpad.net/">LinqPad</a> in this case)...<br />
<br />
<pre class="brush: csharp">private void Insert()
{
var repo = new ProjectRepository();
var project = new Project()
{
Owner = Guid.Parse("8ad82668-4b08-49c9-87ef-80870bfb4b85");
Name = "My new project",
StartDate = DateTime.Now,
Status = 4,
Tasks = new List<Task>()
{
new Task { Name = "Task 1", IsCompleted = true },
new Task { Name = "Task 2" }
}
};
repo.Insert(project);
}
private void List()
{
var repo = new ProjectRepository();
var projects = repo.List("static");
projects.Dump();
}
private void Load()
{
var repo = new ProjectRepository();
var project = repo.Load("8ad82668-4b08-49c9-87ef-80870bfb4b85", "c7d5f59c-72da-48de-83ca-265d8609ec02");
project.Dump();
}
private void Update()
{
var repo = new ProjectRepository();
var project = repo.Load("8ad82668-4b08-49c9-87ef-80870bfb4b85", "c7d5f59c-72da-48de-83ca-265d8609ec02");
project.Name = "Modified name " + DateTime.Now.Ticks;
repo.Update(project);
}
private void Delete()
{
var repo = new ProjectRepository();
var project = repo.Load("8ad82668-4b08-49c9-87ef-80870bfb4b85", "c7d5f59c-72da-48de-83ca-265d8609ec02");
repo.Delete(project);
}
private void DeleteDirectly()
{
var repo = new ProjectRepository();
repo.Delete("8ad82668-4b08-49c9-87ef-80870bfb4b85", "c7d5f59c-72da-48de-83ca-265d8609ec02");
}
</pre><br />
What I've shown you here is really basic and we do have some limitations like the fact that a serialized document can't be bigger than 64KB in size and we are limited to 251 extra properties (or indexes). Still, it is a good start for a prototype of a document store.<br />
<br />
I'm currently working on a more self-contained library to help me use the Azure Table Storage Service as a Document Store. More on this in a future post.<br />
<br />
If you want you can grab all the code (Gist) on GitHub <a href="https://gist.github.com/plaurin/5260045">here</a> and <a href="https://gist.github.com/plaurin/5260074">here</a>.<br />
<br />
<h3>See also</h3>- <a href="http://pascallaurin42.blogspot.com/2013/03/using-azure-table-storage-with-dynamic.html">Using Azure Table Storage with dynamic table entities</a><br />
- <a href="http://pascallaurin42.blogspot.com/2013/05/using-azure-blob-storage-to-store.html">Using Azure Blob Storage to store documents</a><br />
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com6tag:blogger.com,1999:blog-5724958397009584147.post-44630655636942566112013-03-12T21:52:00.000-04:002013-05-09T23:02:41.518-04:00Using Azure Table Storage with dynamic table entitiesI've been working with <a href="http://www.windowsazure.com/">Windows Azure</a> for a few months now and I was trying to figure out a way to use the <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/">Azure Table Storage Service</a> with POCOs and complex types rather than only classes inheriting from the <a href="http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.table.tableentity.aspx">TableEntity</a> base class. Turns out that the only thing the <a href="http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.table.cloudtableclient.aspx">CloudTableClient</a> cares about is the <a href="http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.table.itableentity">ITableEntity</a> interface. <a href="http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.table.dynamictableentity.aspx">DynamicTableEntity</a> also implements ITableEntity but is only used for querying and updating entities. You can see it in action in the example on how to <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/#header-14">query only a subset of an entity's properties</a>.<br />
<br />
So I started to wonder if it was possible to create class that implements <a href="http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.table.itableentity">ITableEntity</a> and offer the dynamic features of an <a href="http://blogs.msdn.com/b/csharpfaq/archive/2009/10/01/dynamic-in-c-4-0-introducing-the-expandoobject.aspx">ExpandoObject</a>. After a bit of hacking around in <a href="http://www.linqpad.net/">LinqPad</a> I have this solution.<br />
<br />
<script src="https://gist.github.com/plaurin/5148574.js"></script><br />
<noscript><pre>public class ElasticTableEntity : DynamicObject, ITableEntity,
ICustomMemberProvider // For LinqPad's Dump
{
public ElasticTableEntity()
{
this.Properties = new Dictionary<string, EntityProperty>();
}
public IDictionary<string, EntityProperty> Properties { get; private set; }
public object this[string key]
{
get
{
if (!this.Properties.ContainsKey(key))
this.Properties.Add(key, this.GetEntityProperty(key, null));
return this.Properties[key];
}
set
{
var property = this.GetEntityProperty(key, value);
if (this.Properties.ContainsKey(key))
this.Properties[key] = property;
else
this.Properties.Add(key, property);
}
}
#region DynamicObject overrides
public override bool TryGetMember(GetMemberBinder binder, out object result)
{
result = this[binder.Name];
return true;
}
public override bool TrySetMember(SetMemberBinder binder, object value)
{
this[binder.Name] = value;
return true;
}
#endregion
#region ITableEntity implementation
public string PartitionKey { get; set; }
public string RowKey { get; set; }
public DateTimeOffset Timestamp { get; set; }
public string ETag { get; set; }
public void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext)
{
this.Properties = properties;
}
public IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext)
{
return this.Properties;
}
#endregion
#region ICustomMemberProvider implementation for LinqPad's Dump
public IEnumerable<string> GetNames()
{
return new[] { "PartitionKey", "RowKey", "Timestamp", "ETag" }
.Union(this.Properties.Keys);
}
public IEnumerable<Type> GetTypes()
{
return new[] { typeof(string), typeof(string), typeof(DateTimeOffset), typeof(string) }
.Union(this.Properties.Values.Select(x => this.GetType(x.PropertyType)));
}
public IEnumerable<object> GetValues()
{
return new object[] { this.PartitionKey, this.RowKey, this.Timestamp, this.ETag }
.Union(this.Properties.Values.Select(x => this.GetValue(x)));
}
#endregion
private EntityProperty GetEntityProperty(string key, object value)
{
if (value == null) return new EntityProperty((string)null);
if (value.GetType() == typeof(byte[])) return new EntityProperty((byte[])value);
if (value.GetType() == typeof(bool)) return new EntityProperty((bool)value);
if (value.GetType() == typeof(DateTimeOffset)) return new EntityProperty((DateTimeOffset)value);
if (value.GetType() == typeof(DateTime)) return new EntityProperty((DateTime)value);
if (value.GetType() == typeof(double)) return new EntityProperty((double)value);
if (value.GetType() == typeof(Guid)) return new EntityProperty((Guid)value);
if (value.GetType() == typeof(int)) return new EntityProperty((int)value);
if (value.GetType() == typeof(long)) return new EntityProperty((long)value);
if (value.GetType() == typeof(string)) return new EntityProperty((string)value);
throw new Exception("not supported " + value.GetType() + " for " + key);
}
private Type GetType(EdmType edmType)
{
switch (edmType)
{
case EdmType.Binary : return typeof(byte[]);
case EdmType.Boolean : return typeof(bool);
case EdmType.DateTime : return typeof(DateTime);
case EdmType.Double : return typeof(double);
case EdmType.Guid : return typeof(Guid);
case EdmType.Int32 : return typeof(int);
case EdmType.Int64 : return typeof(long);
case EdmType.String : return typeof(string);
default: throw new Exception("not supported " + edmType);
}
}
private object GetValue(EntityProperty property)
{
switch (property.PropertyType)
{
case EdmType.Binary : return property.BinaryValue;
case EdmType.Boolean : return property.BooleanValue;
case EdmType.DateTime : return property.DateTimeOffsetValue;
case EdmType.Double : return property.DoubleValue;
case EdmType.Guid : return property.GuidValue;
case EdmType.Int32 : return property.Int32Value;
case EdmType.Int64 : return property.Int64Value;
case EdmType.String : return property.StringValue;
default: throw new Exception("not supported " + property.PropertyType);
}
}
}</pre></noscript><br />
In this snippet I also implemented the <a href="http://www.linqpad.net/FAQ.aspx#extensibility"><span id="goog_937116546"></span>ICustomMemberProvider<span id="goog_937116547"></span></a> which is part of the LinqPad extensions API for queries (more on this <a href="http://www.linqpad.net/FAQ.aspx#extensibility">here</a>). In Visual Studio we'll need to remove that code.<br />
<br />
We can now use the <a href="https://gist.github.com/plaurin/5148574">ElasticTableEntity</a> class like this:<br />
<br />
<script src="https://gist.github.com/plaurin/5148570.js"></script><br />
<noscript><pre>var connectionString="...";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
var client = storageAccount.CreateCloudTableClient();
var table = client.GetTableReference("Demo");
table.CreateIfNotExists();
// dynamic keyword to use a dynamic entity
dynamic entity = new ElasticTableEntity();
entity.PartitionKey = "Partition123";
entity.RowKey = (DateTime.MaxValue.Ticks - DateTime.Now.Ticks).ToString();
entity.Name = "Pascal";
entity.Number = 34;
entity.Bool = false;
entity.Date = new DateTime(1912, 3, 4);
entity.TokenId = Guid.NewGuid();
entity["LastName"] = "Laurin";
// Insert the entity we created dynamically
table.Execute(TableOperation.Insert(entity));
// Query all entities in the table
var query = new TableQuery<ElasticTableEntity>();
var result = table.ExecuteQuery(query)
.ToList().Dump("Result");
// Query only a subset of properties
var result2 = table.ExecuteQuery(query.Select(new[] { "FirstName", "Date" }))
.ToList().Dump("Result with projection");</pre></noscript><br />
Please note that you need to use the dynamic keyword to be able to define properties dynamically. You can also use the entity indexer like I did with the LastName property.<br />
<br />
<div style="overflow-x: scroll;"><table class="headingpresenter" style="border-collapse: collapse;border: none;border-top: 1px;margin: 1em 0 1.2em 0.15em;border-left: 3px dotted #1a5"><tr><th class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0.2em 0.5em;margin: 0;text-align: left;background-color: white;font-family: Arial;font-size: 110%;font-weight: bold;color: green">Result</th></tr>
<tr><td class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0 0.6em;margin: 0"><table id="t2" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="10" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0 0.1em;margin: 0;font-family: tahoma;font-size: 100%;font-weight: bold;background-color: #17b;color: white">List<ElasticTableEntity> (1 item)</td></tr>
<tr><th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">PartitionKey</th><th title="System.DateTimeOffset" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">RowKey</th><th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Timestamp</th><th title="System.Boolean" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">ETag</th><th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">FirstName</th><th title="System.Guid" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Number</th><th title="System.Guid" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Bool</th><th title="System.Guid" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Date</th><th title="System.Guid" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">TokenId</th><th title="System.Guid" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">LastName</th></tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Partition123</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">2520391787589766073</td><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">2013-03-13 1:00:40 AM +00:00</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">W/"datetime'2013-03-13T01%3A00%3A40.619873Z'"</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Pascal</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">34</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">False</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">1912-03-04 12:00:00 AM +00:00</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">50604c02-f01c-48fc-862e-7ea66153f434</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Laurin</td></tr>
</table></td></tr>
</table></div><div><table class="headingpresenter" style="border-collapse: collapse;border: none;border-top: 1px;margin: 1em 0 1.2em 0.15em;border-left: 3px dotted #1a5"><tr><th class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0.2em 0.5em;margin: 0;text-align: left;background-color: white;font-family: Arial;font-size: 110%;font-weight: bold;color: green">Result with projection</th></tr>
<tr><td class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0 0.6em;margin: 0"><table id="t4" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="6" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0 0.1em;margin: 0;font-family: tahoma;font-size: 100%;font-weight: bold;background-color: #17b;color: white">List<ElasticTableEntity> (1 item)</td></tr>
<tr><th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">PartitionKey</th><th title="System.DateTimeOffset" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">RowKey</th><th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Timestamp</th><th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">ETag</th><th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Date</th><th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">FirstName</th></tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Partition123</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">2520391787589766073</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">2013-03-13 1:00:40 AM +00:00</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">W/"datetime'2013-03-13T01%3A00%3A40.619873Z'"</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">1912-03-04 12:00:00 AM +00:00</td><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Pascal</td></tr>
</table></td></tr>
</table></div><br />
The <a href="https://gist.github.com/plaurin/5148574">ElasticTableEntity</a> allows us to define properties at run time which will be added to the table when inserting the entities. Tables in the Azure Table Storage have flexible schema so we are free to store entities with different properties <a href="http://msdn.microsoft.com/en-us/library/dd179338.aspx">as long a we respect some limitations</a>:<br />
<ul><li>Entities can have no more than 252 different properties (that's for the Table)</li>
<li>An Entity's data can be up to 1 MB in size</li>
<li>A property must be one of the following types : byte[], bool, DateTime, double, Guid, int, long or string</li>
<li>A property value can be up to 64 KB in size (for string and byte array)</li>
<li>A property name is case sensitive and can be no more than 255 characters in length</li>
</ul><div>You can store about any kind of data as long as it is one of the supported data type. You could also encode other kind of date type in a byte array or a string (like a json document). Just be careful to always stick to one data type for a property (yes, we can store like int, bool and string in the same column using different entities!)</div><div><br />
</div><div>That's it for now. Next time I'll show you how to use the <a href="http://www.windowsazure.com/en-us/develop/net/how-to-guides/table-services/">Windows Azure Table Storage Service</a> as a <a href="http://en.wikipedia.org/wiki/Document-oriented_database">document-oriented database</a> with the ElasticTableEntity.</div><br />
<br />
<br />
<h3>See also</h3>- <a href="http://pascallaurin42.blogspot.ca/2013/04/document-oriented-database-with-azure.html">Document oriented database with Azure Table Storage Service</a> <br />
- <a href="http://pascallaurin42.blogspot.com/2013/05/using-azure-blob-storage-to-store.html">Using Azure Blob Storage to store documents</a><br />
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com14tag:blogger.com,1999:blog-5724958397009584147.post-43908254298662822912013-02-03T16:21:00.000-05:002013-02-07T12:07:12.468-05:00Using MiniProfiler for Entity FrameworkThe most important thing to know when using an ORM for your database querying is what exactly happens behind the scene. Do you know if what you are doing will produce acceptable SQL? Do you know if the query will execute fast? Worst of all, do you know how many queries will be executed?<br />
<br />
To help answer those questions we need to profile our ORM. Good tools like <a href="http://msdn.microsoft.com/en-us/library/ms181091.aspx">SQL Profiler</a>, <a href="http://www.hibernatingrhinos.com/products/efprof">Entity Framework Profiler</a> and <a href="http://www.hibernatingrhinos.com/products/NHProf">NHibernate Profiler</a> exists but you need to spend money for them.<br />
<br />
On the free side of things, <a href="http://miniprofiler.com/">MiniProfiler</a> is a library we can add to our project via a <a href="http://nuget.org/packages/MiniProfiler.EF/">NuGet package</a> for Entity Framework in this case. MiniProfiler was created for ASP.Net MVC applications in mind but we can still use it in a desktop application using another library called <a href="http://nuget.org/packages/MiniProfiler.Windows/">MiniProfiler.Window</a>.<br />
<br />
Here is a small example on how to use both libraries together:<br />
<br />
<script src="https://gist.github.com/4703605.js"></script><br />
<noscript><pre>MiniProfilerEF.Initialize_EF42(); // When using Code First
ConsoleProfiling.Start();
using (StackExchange.Profiling.MiniProfiler.Current.Step("StepName"))
{
using (var context = new Context())
{
var result = context.Entities
.Where(p => p.Property.StartsWith("Value"))
.ToList();
}
}
foreach (var sqlTiming in ConsoleProfiling.StopAndGetProfiler().GetSqlTimings())
{
Console.WriteLine("Duration: " + sqlTiming.DurationMilliseconds);
Console.WriteLine(sqlTiming.CommandString);
if (sqlTiming.Parameters != null)
{
foreach (var parameter in sqlTiming.Parameters)
Console.WriteLine(parameter.Name + ": " + parameter.Value);
}
Console.WriteLine("------------------");
}
</pre></noscript><br />
<br />
This will output to the console the duration of the execution, the raw SQL and parameters of the query. Of course we'll want to refactor this code a bit but it gives you an idea on how MiniProfiler works.<br />
<br />
Just a small warning, for reasons unknown to me mixing MiniProfiler and Entity Framework's database initialization like DropCreateDatabaseAlways is not working. As long as we are disabling the initialization with Database.SetInitializer<Context>(<span style=' color: Blue;'>null</span>) everything's working just fine.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com1tag:blogger.com,1999:blog-5724958397009584147.post-72423012157436833872012-11-01T11:43:00.002-04:002012-11-01T11:43:40.839-04:00Querying TFS using Roslyn<a href="http://msdn.microsoft.com/en-us/vstudio/roslyn.aspx">Roslyn</a> is an ambitious new project by Microsoft. It is the fully written in .Net replacement of the old compilers used in Visual Studio and should be shipped in a future version of the IDE. The current compilers csc.exe and vbc.exe were first shipped in version 1.0 of .Net more than 10 years ago.<br />
<br />
Roslyn offers a lot of new possibilities. As of today Roslyn in still is in CTP and the latest release was in September 2012. For this first post on Roslyn I want to explore the new C# Interactive window (you will need to <a href="http://www.microsoft.com/en-us/download/details.aspx?id=34685">download</a> and install the CTP for that).<br />
<br />
For this demonstration I'll use one of my previous query for LinqPad, <a href="http://pascallaurin42.blogspot.ca/2012/08/tfs-queries-searching-in-all-work-items.html">Searching in all work items</a>.<br />
<br />
First we need to open the C# Interactive window (from menu VIEW -> Other Windows -> C# Interactive)<br />
<br />
<img border="0" src="http://2.bp.blogspot.com/-nj6aXBTO_qM/UIvgCVPD6-I/AAAAAAAACtE/6GZ8BqiLoZs/s1600/Roslyn9.png" /><br />
<br />
Then we start by referencing the required libraries using the special <b>#r</b> instruction<br />
<pre class="brush: csharp">> #r "Microsoft.TeamFoundation.Client"
> #r "Microsoft.TeamFoundation.Common"
> #r "Microsoft.TeamFoundation.VersionControl.Client"
> #r "Microsoft.TeamFoundation.WorkItemTracking.Client"</pre><br />
Next we need to import the namespaces we are going to use with the using statement like this<br />
<pre class="brush: csharp">using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.WorkItemTracking.Client;</pre><br />
Now we are ready to write some real code. I'll be using my online Team Foundation Service account for this. You can register for a free account <a href="https://tfs.visualstudio.com/">here</a>.<br />
<pre class="brush: csharp">> var uri = new Uri("https://my.visualstudio.com/DefaultCollection/");
> var tfs = TfsTeamProjectCollectionFactory
.GetTeamProjectCollection(uri);
> tfs.EnsureAuthenticated();
> var workItemStore = tfs.GetService<WorkItemStore>();
> var title = "phone";
> var query = string.Format(@"
Select [Id], [Work Item Type], [Title], [State]
Where [Title] Contains '{0}'
From WorkItems",
title);
> var results = workItemStore.Query(query).Cast<WorkItem>()
.Select(wi => new
{
wi.Id,
wi.AreaPath,
Type = wi.Type.Name,
wi.Title,
wi.State
})
.OrderBy(wi => wi.AreaPath).ThenBy(wi => wi.Type).ThenBy(wi => wi.Title);</pre><br />
Of course we don't have access to LinqPad's Dump function so we have to print the results ourself.<br />
<pre class="brush: csharp">> foreach (var r in results)
{
Console.WriteLine("{0} - {1} [{2}] {3} <{4}>",
r.Id, r.AreaPath, r.Type, r.Title, r.State);
}
2 - Project1 [Bug] Bug #1 - Fix the phone number on the Contact page <New></pre><br />
I think using LinqPad is still easier for this kind of work but those who don't have a paid version of LinqPad will like the facts that Roslyn is free, offers a way to write queries using Intellisense and also a more interactive way to work and find the data you are looking for.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-86580884289192665232012-10-07T17:39:00.000-04:002013-02-07T12:11:15.471-05:00How to reference and reuse LinqPad queriesOne missing feature from LinqPad is the ability to easily reuse code in a query we wrote in other queries.<br />
<br />
If we really want to do it we have the following options:<br />
<ul><li>Copy and paste the code into the new query</li>
<li>Open Visual Studio, create a project, create a class and copy the code you want to reuse. Compile the project and finally reference the generated dll from your query.</li>
<li>Compile the LinqPad query directly (using a special script), then reference the generated dll from your query.</li>
</ul><br />
Today we will look at the third option. Found in the .Net framework is a very interesting API called the CodeDOM API. This API can be used to compile .Net code at run-time, like the compiler does. With this, we will be able to parse a LinqPad query and compile it.<br />
<br />
This is what we will look into right now.<br />
<br />
<h3>Compiling a LinqPad query</h3><br />
Those are the steps we need to do to accomplish query compilation:<br />
<ol><li>Read the query and extract the options, usings and references</li>
<li>Create the CodeDOM objects and set the options and references</li>
<li>Create a string of the code file including the usings and wrap the query code inside a class</li>
<li>Compile the code</li>
</ol><br />
I'll explain in details each steps in future blog posts but for now I'll give you my query to compile LinqPad queries. You will notice that this is a self compiling query as the query compile itself to a dll I can reuse it to compile other queries!<br />
<br />
<img border="0" height="59" src="http://3.bp.blogspot.com/-cz90HZ9y5ag/UHHykBNHUfI/AAAAAAAACsw/cQtqkHg1xTc/s320/Achievement.jpg" width="320" /><br />
<br />
<br />
You can figure out the Compiler class usage from the Main function for now.<br />
<br />
<h3>Additional Namespace Imports</h3>System.CodeDom.Compiler<br />
System.Runtime.InteropServices<br />
Microsoft.CSharp<br />
<br />
<h3>Query (C# Program)</h3><br />
<script src="https://gist.github.com/3849578.js?file=SelfCompile"></script><br />
<script src="https://gist.github.com/3849578.js"></script><br />
<br />
<noscript><pre>void Main()
{
Environment.CurrentDirectory = Path.GetDirectoryName(Util.CurrentQueryPath);
Compiler.CompileFiles(Options.CreateOnDiskDll(
@namespace: "LinqPad.QueriesCompiler",
outputFile: "LinqPad.QueriesCompiler.dll")
.AddCodeFile(CodeType.LinqProgramTypes, Util.CurrentQueryPath))
.Dump("Successfully created assembly at " + DateTime.Now.ToLocalTime());
}
// Define other methods and classes here
public static class Compiler
{
public static Assembly CompileFiles(Options options)
{
var outputPath = options.IsInMemory ? "" : options.OutputFile;
if (!options.CodeFiles.Any())
throw new InvalidOperationException("Should add at least one file to compile.");
foreach (var codeFile in options.CodeFiles)
{
codeFile.RawContent = File.ReadAllLines(codeFile.FilePath);
codeFile.Query = GetQuery(new FileInfo(codeFile.FilePath).DirectoryName, codeFile.RawContent);
GetCode(codeFile, options.Namespace);
}
return BuildAssembly(options.CodeFiles, options, outputPath);
}
public static Query GetQuery(string folder, IEnumerable<string> content)
{
var xml = string.Join("\r\n", content.TakeWhile(l => l.Trim().StartsWith("<")));
var queryElement = XDocument.Parse(xml).Element("Query");
if (queryElement == null) throw new InvalidOperationException("Missing <Query> header definition");
var query = new Query
{
Kind = queryElement.Attribute("Kind").Value,
Namespaces = queryElement.Elements("Namespace").Select(n => n.Value).ToList(),
GACReferences = queryElement.Elements("GACReference").Select(n => n.Value).ToList(),
RelativeReferences = queryElement.Elements("Reference").Where(e => e.Attribute("Relative") != null)
.Select(n => n.Attribute("Relative").Value)
.Select(x => new FileInfo(Path.Combine(folder, x)).FullName)
.ToList(),
OtherReferences = queryElement.Elements("Reference").Where(e => e.Attribute("Relative") == null)
.Select(n => n.Value.Replace("<RuntimeDirectory>", RuntimeEnvironment.GetRuntimeDirectory()))
.Select(n => n.Replace("<ProgramFilesX86>", Environment.GetFolderPath(Environment.SpecialFolder.ProgramFilesX86)))
.ToList()
};
return query;
}
public static IEnumerable<string> GetCode(IEnumerable<IEnumerable<string>> contents, Query query, string ns)
{
return contents.Select(c => GetCode(c, query, ns));
}
private static void GetCode(CodeFile codeFile, string @namespace)
{
IEnumerable<string> result = null;
if (codeFile.Query.Kind != "Program" && codeFile.Query.Kind != "Statements")
throw new InvalidOperationException("Only queries of type C# program and C# statements are supported");
var filteredContent = codeFile.RawContent.SkipWhile(l => l.Trim().StartsWith("<"));
switch (codeFile.Type)
{
case CodeType.LinqStatements:
result = WrapInClass(WrapInMain(filteredContent));
break;
case CodeType.LinqProgramTypes:
result = FilterTypes(filteredContent);
break;
case CodeType.LinqProgram:
result = WrapInClass(filteredContent);
break;
default:
throw new InvalidOperationException("Only queries of type C# program and C# statements are supported");
}
codeFile.Content = GetCode(result, codeFile.Query, @namespace);
}
private static IEnumerable<string> WrapInClass(IEnumerable<string> inputCode)
{
var s = new[] { "public class Program {" };
var e = new[] { "}" };
return s.Concat(inputCode).Concat(e);
}
private static IEnumerable<string> WrapInMain(IEnumerable<string> inputCode)
{
var s = new[] { "public static void Main() {" };
var e = new[] { "}" };
return s.Concat(inputCode).Concat(e);
}
private static IEnumerable<string> FilterTypes(IEnumerable<string> inputCode)
{
return inputCode.SkipWhile(l => l.Trim() != "// Define other methods and classes here");
}
public static string GetCode(IEnumerable<string> content, Query query, string ns)
{
var code = string.Join(Environment.NewLine, content);
var codeBuilder = new StringBuilder();
codeBuilder.AppendLine("using " + string.Join(";\r\nusing ", query.Namespaces.Union(StandardNamespaces)) + ";");
codeBuilder.AppendLine(string.Format("namespace {0} {{", ns));
codeBuilder.AppendLine(code);
codeBuilder.AppendLine("}");
return codeBuilder.ToString();
}
public static Assembly BuildAssembly(IEnumerable<CodeFile> codeFiles, Options options, string outputPath)
{
var providerOptions = new Dictionary<string, string> { { "CompilerVersion", "v4.0" } };
var provider = new CSharpCodeProvider(providerOptions);
var assemblies = new[]
{
codeFiles.SelectMany(c => c.Query.GACReferences.Select(s => Assembly.Load(s).Location)),
codeFiles.SelectMany(c => c.Query.RelativeReferences.Select(s => Assembly.LoadFrom(s).Location)),
codeFiles.SelectMany(c => c.Query.OtherReferences),
Assemblies
};
var compilerparams = new CompilerParameters
{
GenerateExecutable = !string.IsNullOrWhiteSpace(options.StartupObject),
OutputAssembly = options.IsInMemory ? null : outputPath,
GenerateInMemory = true,
IncludeDebugInformation = true,
MainClass = options.StartupObject
};
compilerparams.ReferencedAssemblies.AddRange(assemblies.SelectMany(a => a).ToArray());
var results = provider.CompileAssemblyFromSource(compilerparams, codeFiles.Select(f => f.Content).ToArray());
if (results.Errors.HasErrors)
{
var errors = new StringBuilder("Compiler Errors:\r\n");
foreach (CompilerError error in results.Errors)
{
errors.AppendFormat("File {0}, Line {1},{2}\t: {3}\r\n",
error.FileName, error.Line, error.Column, error.ErrorText);
}
throw new Exception("Errors compiling:\r\n" + errors + "\r\n\r\n" +
string.Join(Environment.NewLine, codeFiles.Select(f => f.FilePath + ":\r\n" + AddLineNumber(f.Content))));
}
return results.CompiledAssembly;
}
private static string AddLineNumber(string code)
{
var lines = code.Split(new[] { Environment.NewLine }, StringSplitOptions.None);
return string.Join(Environment.NewLine, lines.Select ((x, i) => (i + 1).ToString().PadLeft(4) + ": " + x));
}
private static List<string> StandardNamespaces
{
get
{
return new List<string>
{
"System",
"System.IO",
"System.Text",
"System.Text.RegularExpressions",
"System.Diagnostics",
"System.Threading",
"System.Reflection",
"System.Collections",
"System.Collections.Generic",
"System.Linq",
"System.Linq.Expressions",
"System.Data",
"System.Data.SqlClient",
"System.Data.Linq",
"System.Data.Linq.SqlClient",
"System.Xml",
"System.Xml.Linq",
"System.Xml.XPath"
};
}
}
private static List<string> Assemblies
{
get
{
return new List<string>
{
"System.dll",
"System.Core.dll",
"System.Data.dll",
"System.Xml.dll",
"System.Xml.Linq.dll",
"System.Data.Linq.dll",
"System.Drawing.dll",
"System.Data.DataSetExtensions.dll"
};
}
}
}
public class Query
{
public string Kind { get; set; }
public List<string> Namespaces { get; set; }
public List<string> GACReferences { get; set; }
public List<string> RelativeReferences { get; set; }
public List<string> OtherReferences { get; set; }
}
public enum CodeType
{
LinqStatements, // Wrap inside class and Main() method
LinqProgramTypes, // Code after 'Define other methods and classes here' directly inside namespace
LinqProgram, // Wrap all code inside class
}
public class CodeFile
{
public CodeType Type { get; set; }
public string FilePath { get; set; }
public string[] RawContent { get; set; }
public Query Query { get; set; }
public string Content { get; set; }
}
public class Options
{
private Options(string @namespace)
{
if (string.IsNullOrWhiteSpace(@namespace)) throw new ArgumentNullException("namespace");
this.CodeFiles = new List<CodeFile>();
this.Namespace = @namespace;
}
public static Options CreateInMemoryDll(string @namespace)
{
var options = new Options(@namespace);
options.OutputFile = null;
options.IsInMemory = true;
options.StartupObject = null;
return options;
}
public static Options CreateOnDiskDll(string @namespace, string outputFile)
{
if (string.IsNullOrWhiteSpace(outputFile)) throw new ArgumentNullException("outputFile");
var options = new Options(@namespace);
options.OutputFile = outputFile;
options.IsInMemory = false;
options.StartupObject = null;
return options;
}
public static Options CreateOnDiskExe(string @namespace, string outputFile, string startupObject)
{
if (string.IsNullOrWhiteSpace(outputFile)) throw new ArgumentNullException("outputFile");
if (string.IsNullOrWhiteSpace(startupObject)) throw new ArgumentNullException("startupObject");
var options = new Options(@namespace);
options.OutputFile = outputFile;
options.IsInMemory = false;
options.StartupObject = startupObject;
return options;
}
public Options AddCodeFile(CodeType type, string filePath)
{
this.CodeFiles.Add(new CodeFile { Type = type, FilePath = filePath });
return this;
}
public string Namespace { get; private set; }
public string OutputFile { get; private set; }
public List<CodeFile> CodeFiles { get; private set; }
public string StartupObject { get; private set; }
public bool IsInMemory { get; private set; }
}
</pre></noscript>Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com1tag:blogger.com,1999:blog-5724958397009584147.post-78106859292974192712012-09-18T21:04:00.000-04:002012-09-18T21:04:30.662-04:00Using NuGet with LinqPad for freeRecently <a href="http://pascallaurin42.blogspot.ca/2012/06/nuget-integration-in-linqpad.html">LinqPad got a great integration with NuGet</a> in the paid version of the software<br />
<br />
Unfortunately for the user of the free version it is a bit more difficult to use NuGet but it's not impossible.<br />
<br />
Today I'm going to show you a way to use NuGet to get libraries into your LinqPad queries.<br />
<br />
<h3>NuGet Package Explorer</h3><br />
You can download the <a href="http://nuget.codeplex.com/releases/view/59864">NuGet Package Explorer from CodePlex</a><br />
<br />
The primary use for this tool is to explorer, edit and publish NuGet packages. You could also use this tool to extract the files contained in the packages and put them in a folder, and that's what we'll do now.<br />
<br />
First we need to <b>Open a package from online feed</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-RPIn2toopA0/UFe0VCKfo1I/AAAAAAAACrw/uNQnkOmI_qw/s1600/NPE01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-RPIn2toopA0/UFe0VCKfo1I/AAAAAAAACrw/uNQnkOmI_qw/s1600/NPE01.png" /></a></div><br />
Here we could query for the package we want.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-UxdHxuyBIss/UFe0VSuAcQI/AAAAAAAACr4/_gFlce7ATPY/s1600/NPE02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-UxdHxuyBIss/UFe0VSuAcQI/AAAAAAAACr4/_gFlce7ATPY/s1600/NPE02.png" /></a></div><br />
Select a package and click OK. We can now take a peek inside the package if we want. For us, we want to <b>Export</b> the content from the <b>File</b> menu.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-ZjsF35kNZrw/UFe0V7uCyAI/AAAAAAAACsA/k8u3FP_Xde4/s1600/NPE03.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-ZjsF35kNZrw/UFe0V7uCyAI/AAAAAAAACsA/k8u3FP_Xde4/s1600/NPE03.png" /></a></div><br />
You need choose a folder where to export the files of the package. Finally, in LinqPad we need to add a reference to the DLL by going in the <b>Query Properties (F4)</b> and clicking <b>Browse</b>. After that we are ready to start writing our queries.<br />
<br />
<h3>Using the NuGet local cache folder</h3><br />
One another tip I've got for you is to add the local NuGet cache folder to the list of <i>online feed</i> as a shortcut for when we want to access a package we previously downloaded with the NuGet Package Explorer or even Visual Studio.<br />
<br />
First, from the <b>Tools</b> menu select <b>View NuGet download cache</b>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-qV3-zmZnsFU/UFe0WQ0vFjI/AAAAAAAACsI/4hf7BDnatOk/s1600/NPE04.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-qV3-zmZnsFU/UFe0WQ0vFjI/AAAAAAAACsI/4hf7BDnatOk/s1600/NPE04.png" /></a></div><br />
This will open a Windows Explorer on the folder <i>%AppData%\Local\NuGet\Cache</i>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-bS_FX78HpHI/UFe0Wqb3xDI/AAAAAAAACsQ/5lBvWKbPVSM/s1600/NPE05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-bS_FX78HpHI/UFe0Wqb3xDI/AAAAAAAACsQ/5lBvWKbPVSM/s1600/NPE05.png" /></a></div><br />
Now copy this path, go back to NuGet Package Explorer and <b>Open a package from online feed</b>. Then paste the path into <b>Package source</b> field and click <b>Reload</b>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-IQI-FEQvG1Y/UFe0XI2T4uI/AAAAAAAACsY/LEP9n8FUuzw/s1600/NPE06.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-IQI-FEQvG1Y/UFe0XI2T4uI/AAAAAAAACsY/LEP9n8FUuzw/s1600/NPE06.png" /></a></div><br />
From now on you will be able to choose between the official NuGet.org feed and your local cache when you want to open NuGet packages.<br />
<br />
<h3>NuGet.exe</h3><br />
Of course for the hardcore user of the command line interface one could use nuget.exe directly to download and extract the files rather than using the Package Explorer. I'm definitely not that hardcore. :-)<br />
<br />
NuGet.exe is also available from the <a href="http://nuget.codeplex.com/releases/view/58939">CodePlex site</a>.<br />
Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-48249868113857948522012-08-31T23:18:00.000-04:002012-10-30T14:41:52.840-04:00Using log4net MemoryAppender for unit testingSometimes unit testing existing code can be really hard. Especially when coding by exceptions (using Try.. Catch.. then do nothing!).<br />
<br />
This will look a bit like this<br />
<br />
<script src="https://gist.github.com/3563082.js?file=TryCache.cs"></script><br />
<noscript><pre>public void DoSomething()
{
try
{
ReallyDoSomething();
}
catch (Exception ex)
{
this.logger.LogError("An error occured!", ex);
}
}</pre></noscript><br />
From the outside we might not be able to test whether ReallyDoSomething has thrown an exception except if we can validate state changes done by ReallyDoSomething. And if we can't change the code much (like adding dependencies using parameters or change the return type of the method) we won't be able to write that test at all.<br />
<br />
Another way is to use the logger to validate that.<br />
<br />
The piece of code above use <a href="http://logging.apache.org/log4net/"
>Log4Net</a> as the logging framework and allow us to use it in our unit tests like this<br />
<br />
<script src="https://gist.github.com/3563082.js?file=MemoryAppenderUT.cs"></script><br />
<noscript><pre>[SetUp]
public void SetUp()
{
this.memoryAppender = new MemoryAppender();
BasicConfigurator.Configure(this.memoryAppender);
...
}
[Test]
public void TestSomething()
{
...
// Assert
Assert.IsFalse(
this.memoryAppender.GetEvents().Any(le => le.Level == Level.Error),
"Did not expect any error messages in the logs");
}</pre></noscript><br />
Just like that we can intercept all the logged messages by our production code.<br />
<br />
I've used this trick a few times now when other options were not available. Hope it helps.Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2tag:blogger.com,1999:blog-5724958397009584147.post-5127570422491753222012-08-15T21:07:00.000-04:002012-08-15T21:07:13.695-04:00TFS Queries : Searching in all work itemsFor an intro on LinqPad and the TFS API please read <a href="http://pascallaurin42.blogspot.ca/2012/05/setup-your-new-tfs-linqpad-query.html">this post</a><br />
For the list of all the posts in this series please read <a href="http://pascallaurin42.blogspot.ca/2012/01/tfs-automation-using-linqpad.html">this one</a><br />
<br />
<h3>Context</h3><br />
To search for work items effectively we can use a special TFS API on the WorkItemStore class. The query format is based the Work Item Query Language (or WIQL ) and we can't use Linq directly, but still, we can use LinqPad to write a quick little query.<br />
<br />
<h3>Required references</h3><br />
Microsoft.TeamFoundation.Client<br />
Microsoft.TeamFoundation.Common<br />
Microsoft.TeamFoundation.VersionControl.Client<br />
Microsoft.TeamFoundation.WorkItemTracking.Client<br />
<br />
<h3>Query (C# Statements)</h3><pre class="brush: csharp">var tfs = TfsTeamProjectCollectionFactory
.GetTeamProjectCollection(new Uri("http://localhost:8088/tfs"));
tfs.EnsureAuthenticated();
var workItemStore = tfs.GetService<WorkItemStore>();
var title = "phone";
var query = string.Format(@"
Select [Id], [Work Item Type], [Title], [State]
Where [Title] Contains '{0}'
From WorkItems",
title);
workItemStore.Query(query).Cast<WorkItem>()
.Select(wi => new
{
wi.Id,
wi.AreaPath,
Type = wi.Type.Name,
wi.Title,
wi.State
})
.OrderBy(wi => wi.AreaPath).ThenBy(wi => wi.Type).ThenBy(wi => wi.Title)
.Dump();
</pre><br />
<h3>Result</h3><br />
Here we get the list of all the work items from all projects containing the word 'phone' in their title.<br />
<br />
To find more about WIQL please take a look at the <a href="http://msdn.microsoft.com/en-us/library/bb130306(v=vs.100).aspx">MSDN</a> section on it.<br />
<br />
<div style="overflow-x: scroll;"><div class="spacer" style="margin: 0.6em 0"><table id="t2" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="5" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IOrderedEnumerable<> (1 item)<br />
</td> </tr>
<tr><th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Id</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">AreaPath</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Type</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Title</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">State</th> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">2</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Project1</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Bug</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Fix the phone number on the Contact page</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Done</td> </tr>
</table></div></div>Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com3tag:blogger.com,1999:blog-5724958397009584147.post-45121709727108920522012-08-01T21:04:00.000-04:002012-08-01T21:04:47.746-04:00TFS Queries: Listing all the branchesFor an intro on LinqPad and the TFS API please read <a href="http://pascallaurin42.blogspot.ca/2012/05/setup-your-new-tfs-linqpad-query.html">this post</a><br />
For the list of all the posts in this series please read <a href="http://pascallaurin42.blogspot.ca/2012/01/tfs-automation-using-linqpad.html">this one</a><br />
<br />
<h3>Context</h3><br />
When it comes to branches, using the TFS Web Access or Team Explorer we can navigate around the source code explorer to see branches from various Team Project and related information. But to get the big picture there is an easier way using the TFS API.<br />
<br />
<h3>Required references</h3><br />
Microsoft.TeamFoundation.Client<br />
Microsoft.TeamFoundation.Common<br />
Microsoft.TeamFoundation.VersionControl.Client<br />
<br />
<h3>Query (C# Statements)</h3><pre class="brush: csharp">var tfs = TfsTeamProjectCollectionFactory.
GetTeamProjectCollection(new Uri("http://localhost:8088/tfs"));
tfs.EnsureAuthenticated();
var versionControl = tfs.GetService<VersionControlServer>();
versionControl.QueryRootBranchObjects(RecursionType.Full)
.Where(b => !b.Properties.RootItem.IsDeleted)
.Select(s => new
{
Project = s.Properties.RootItem.Item
.Substring(0, s.Properties.RootItem.Item.IndexOf('/', 2)),
Properties = s.Properties,
DateCreated = s.DateCreated,
ChildBranches = s.ChildBranches
})
.Select(s => new
{
s.Project,
Branch = s.Properties.RootItem.Item.Replace(s.Project, ""),
Parent = s.Properties.ParentBranch != null ?
s.Properties.ParentBranch.Item.Replace(s.Project, "") : "",
Version = (s.Properties.RootItem.Version as ChangesetVersionSpec)
.ChangesetId,
DateCreated = s.DateCreated,
Owner = s.Properties.Owner,
ChildBranches = s.ChildBranches
.Where (cb => !cb.IsDeleted)
.Select(cb => new
{
Branch = cb.Item.Replace(s.Project, ""),
Version = (cb.Version as ChangesetVersionSpec).ChangesetId
})
})
.OrderBy(s => s.Project).ThenByDescending(s => s.Version)
.Dump();
</pre><br />
<h3>Result</h3><br />
Here we can see a bit more information on the branches like related parent and child branches in a simple list.<br />
<br />
<div style="overflow-x: scroll;"><div class="spacer" style="margin: 0.6em 0"><table id="t6" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="7" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IOrderedEnumerable<> (4 items)<br />
</td> </tr>
<tr><th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Project</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Branch</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Parent</th> <th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Version</th> <th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">DateCreated</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Owner</th> <th title="IEnumerable<{System.String,System.Int32}>" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">ChildBranches</th> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">$/Project1</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Release v1.1.0.0</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Main</td> <td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">14</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">04/07/2012 9:18:01 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t7" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="1" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white">IEnumerable<> (0 items)</td> </tr>
</table></td> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">$/Project1</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Main</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"> </td> <td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">8</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">28/06/2012 10:24:05 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t8" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="2" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IEnumerable<> (1 item)<br />
</td> </tr>
<tr><th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Branch</th> <th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Version</th> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Release v1.1.0.0</td> <td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">14</td> </tr>
</table></td> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">$/Project2</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Team</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Main</td> <td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">13</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">04/07/2012 9:17:24 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t9" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="1" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white">IEnumerable<> (0 items)</td> </tr>
</table></td> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">$/Project2</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Main</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"> </td> <td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">12</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">04/07/2012 9:17:02 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t10" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="2" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IEnumerable<> (1 item)<br />
</td> </tr>
<tr><th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Branch</th> <th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Version</th> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">/Team</td> <td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">13</td> </tr>
</table></td> </tr>
<tr><td title="Totals" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right"></td> <td title="Totals" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right"></td> <td title="Totals" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right"></td> <td title="Total=47 Average=11.75" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right">47</td> <td title="Totals" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right"></td> <td title="Totals" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right"></td> <td title="Totals" class="columntotal" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;font-family: Tahoma;background-color: #eee;font-weight: bold;color: #17b;font-size: 90%;text-align: right"></td> </tr>
</table></div></div>Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com0tag:blogger.com,1999:blog-5724958397009584147.post-52635681915420304232012-07-04T22:06:00.000-04:002012-07-04T22:06:23.499-04:00TFS Queries: Generating a changelog from the branch historyFor an intro on LinqPad and the TFS API please read <a href="http://pascallaurin42.blogspot.ca/2012/05/setup-your-new-tfs-linqpad-query.html">this post</a><br />
For the list of all the posts in this series please read <a href="http://pascallaurin42.blogspot.ca/2012/01/tfs-automation-using-linqpad.html">this one</a><br />
<br />
<h3>Context</h3><br />
If you ever need to produce a changelog for a release you could do it by hand, looking at all the work items associated with the changesets or you could use the TFS API to generate it automatically.<br />
<br />
<h3>Required references</h3><br />
Microsoft.TeamFoundation.Client<br />
Microsoft.TeamFoundation.Common<br />
Microsoft.TeamFoundation.VersionControl.Client<br />
Microsoft.TeamFoundation.WorkItemTracking.Client<br />
<br />
<h3>Query (C# Statements)</h3><pre class="brush: csharp">var branch = "$/Project1/Main";
var fromVersion = new ChangesetVersionSpec(1);
var toVersion = VersionSpec.Latest;
var tfs = TfsTeamProjectCollectionFactory
.GetTeamProjectCollection(new Uri("http://localhost:8088/tfs"));
tfs.EnsureAuthenticated();
var versionControl = tfs.GetService<VersionControlServer>();
var history = versionControl.QueryHistory(branch, VersionSpec.Latest, 0,
RecursionType.Full, null, fromVersion, toVersion, int.MaxValue, false,
false, false);
history.OfType<Changeset>()
.Select(x => new
{
x.ChangesetId,
x.CreationDate,
x.Committer,
x.Comment,
WorkItems = x.WorkItems.Select(wi => new
{
wi.Id,
wi.Title,
wi.Description,
wi.State,
wi.Reason
})
})
.Dump("Branch history of: " + branch +
" from " + fromVersion.DisplayString + " to " + toVersion.DisplayString);
history.OfType<Changeset>()
.OrderBy(x => x.ChangesetId)
.SelectMany(x => x.WorkItems)
.Select(wi => string.Format("[{0}] {1} #{2} - {3}",
wi.Reason, wi.Type.Name, wi.Id, wi.Title))
.Dump("Changelog of branch: " + branch +
" from " + fromVersion.DisplayString + " to " + toVersion.DisplayString);
</pre><br />
<h3>Result</h3><br />
With this we will get the list of all the changesets and associated work items from the history of the given branch.<br />
<br />
<div style="overflow-x: scroll;"><table class="headingpresenter" style="border-collapse: collapse;border: none;border-top: 1px;margin: 1em 0 1.2em 0.15em;border-left: 3px dotted #1a5"><tr><th class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0.2em 0.5em;margin: 0;text-align: left;background-color: white;font-family: Arial;font-size: 110%;font-weight: bold;color: green">Branch history of: $/Project1/Main from C1 to T</th> </tr>
<tr><td class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0 0.6em;margin: 0"><br />
<table id="t6" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="5" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IEnumerable<> (4 items)<br />
</td> </tr>
<tr><th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">ChangesetId</th> <th title="System.DateTime" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">CreationDate</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Committer</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Comment</th> <th title="IEnumerable<{System.Int32,System.String,System.String,System.String,System.String}>" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">WorkItems</th> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">11</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">28/06/2012 10:42:14 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Change the phone number</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t7" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="5" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IEnumerable<> (1 item)<br />
</td> </tr>
<tr><th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Id</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Title</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Description</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">State</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Reason</th> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">2</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Fix the phone number on the Contact page</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"> </td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Done</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Work finished</td> </tr>
</table></td> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">10</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">28/06/2012 10:39:58 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Changed the version to 1.1.0.0</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t8" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="5" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IEnumerable<> (1 item)<br />
</td> </tr>
<tr><th title="System.Int32" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Id</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Title</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Description</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">State</th> <th title="System.String" style="vertical-align: top;border: 1px solid #777;padding: 0.1em 0.2em;margin: 0;text-align: left;background-color: #ddd;font-family: tahoma;font-size: 90%;font-weight: bold">Reason</th> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">1</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Change the version number to 1.1.0.0</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"> </td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Done</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Work finished</td> </tr>
</table></td> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">9</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">28/06/2012 10:31:55 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Created a MVC4 Mobile application</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t9" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="1" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white">IEnumerable<> (0 items)</td> </tr>
</table></td> </tr>
<tr><td class="n" style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0;text-align: right">8</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">28/06/2012 10:23:52 PM</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Domain\Pascal</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">Added Main folder for the branch</td> <td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0"><br />
<table id="t10" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="1" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white">IEnumerable<> (0 items)</td> </tr>
</table></td> </tr>
</table></td> </tr>
</table></div><div style="overflow-x: scroll;"><table class="headingpresenter" style="border-collapse: collapse;border: none;border-top: 1px;margin: 1em 0 1.2em 0.15em;border-left: 3px dotted #1a5"><tr><th class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0.2em 0.5em;margin: 0;text-align: left;background-color: white;font-family: Arial;font-size: 110%;font-weight: bold;color: green">Changelog of branch: $/Project1/Main from C1 to T</th> </tr>
<tr><td class="headingpresenter" style="vertical-align: top;border: none;padding: 0 0 0 0.6em;margin: 0"><br />
<table id="t12" style="border-collapse: collapse;border: 2px solid #17b;border-top: 1px;margin: 0.3em 0.2em"><tr><td class="typeheader" colspan="1" style="vertical-align: top;border: 1px solid #aaa;padding: 0 0.2em 0.1em 0.1em;margin: 0;font-family: tahoma;font-size: 90%;font-weight: bold;background-color: #17b;color: white"><br />
IEnumerable<String> (2 items)<br />
</td> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">[Work finished] Task #1 - Change the version number to 1.1.0.0</td> </tr>
<tr><td style="vertical-align: top;border: 1px solid #aaa;padding: 0.1em 0.2em;margin: 0">[Work finished] Bug #2 - Fix the phone number on the Contact page</td> </tr>
</table></td> </tr>
</table></div>Anonymoushttp://www.blogger.com/profile/07583188402446025465noreply@blogger.com2