The Never Ending Journey

Sunday, October 7, 2012

How to reference and reuse LinqPad queries

One missing feature from LinqPad is the ability to easily reuse code in a query we wrote in other queries.

If we really want to do it we have the following options:

Copy and paste the code into the new query
Open Visual Studio, create a project, create a class and copy the code you want to reuse. Compile the project and finally reference the generated dll from your query.
Compile the LinqPad query directly (using a special script), then reference the generated dll from your query.

Today we will look at the third option. Found in the .Net framework is a very interesting API called the CodeDOM API. This API can be used to compile .Net code at run-time, like the compiler does. With this, we will be able to parse a LinqPad query and compile it.

This is what we will look into right now.

Compiling a LinqPad query

Those are the steps we need to do to accomplish query compilation:

Read the query and extract the options, usings and references
Create the CodeDOM objects and set the options and references
Create a string of the code file including the usings and wrap the query code inside a class
Compile the code

I'll explain in details each steps in future blog posts but for now I'll give you my query to compile LinqPad queries. You will notice that this is a self compiling query as the query compile itself to a dll I can reuse it to compile other queries!

You can figure out the Compiler class usage from the Main function for now.

Additional Namespace Imports

System.CodeDom.Compiler
System.Runtime.InteropServices
Microsoft.CSharp

Query (C# Program)

void Main() { Environment.CurrentDirectory = Path.GetDirectoryName(Util.CurrentQueryPath); Compiler.CompileFiles(Options.CreateOnDiskDll( @namespace: "LinqPad.QueriesCompiler", outputFile: "LinqPad.QueriesCompiler.dll") .AddCodeFile(CodeType.LinqProgramTypes, Util.CurrentQueryPath)) .Dump("Successfully created assembly at " + DateTime.Now.ToLocalTime()); } // Define other methods and classes here public static class Compiler { public static Assembly CompileFiles(Options options) { var outputPath = options.IsInMemory ? "" : options.OutputFile; if (!options.CodeFiles.Any()) throw new InvalidOperationException("Should add at least one file to compile."); foreach (var codeFile in options.CodeFiles) { codeFile.RawContent = File.ReadAllLines(codeFile.FilePath); codeFile.Query = GetQuery(new FileInfo(codeFile.FilePath).DirectoryName, codeFile.RawContent); GetCode(codeFile, options.Namespace); } return BuildAssembly(options.CodeFiles, options, outputPath); } public static Query GetQuery(string folder, IEnumerable<string> content) { var xml = string.Join("\r\n", content.TakeWhile(l => l.Trim().StartsWith("<"))); var queryElement = XDocument.Parse(xml).Element("Query"); if (queryElement == null) throw new InvalidOperationException("Missing <Query> header definition"); var query = new Query { Kind = queryElement.Attribute("Kind").Value, Namespaces = queryElement.Elements("Namespace").Select(n => n.Value).ToList(), GACReferences = queryElement.Elements("GACReference").Select(n => n.Value).ToList(), RelativeReferences = queryElement.Elements("Reference").Where(e => e.Attribute("Relative") != null) .Select(n => n.Attribute("Relative").Value) .Select(x => new FileInfo(Path.Combine(folder, x)).FullName) .ToList(), OtherReferences = queryElement.Elements("Reference").Where(e => e.Attribute("Relative") == null) .Select(n => n.Value.Replace("<RuntimeDirectory>", RuntimeEnvironment.GetRuntimeDirectory())) .Select(n => n.Replace("<ProgramFilesX86>", Environment.GetFolderPath(Environment.SpecialFolder.ProgramFilesX86))) .ToList() }; return query; } public static IEnumerable<string> GetCode(IEnumerable<IEnumerable<string>> contents, Query query, string ns) { return contents.Select(c => GetCode(c, query, ns)); } private static void GetCode(CodeFile codeFile, string @namespace) { IEnumerable<string> result = null; if (codeFile.Query.Kind != "Program" && codeFile.Query.Kind != "Statements") throw new InvalidOperationException("Only queries of type C# program and C# statements are supported"); var filteredContent = codeFile.RawContent.SkipWhile(l => l.Trim().StartsWith("<")); switch (codeFile.Type) { case CodeType.LinqStatements: result = WrapInClass(WrapInMain(filteredContent)); break; case CodeType.LinqProgramTypes: result = FilterTypes(filteredContent); break; case CodeType.LinqProgram: result = WrapInClass(filteredContent); break; default: throw new InvalidOperationException("Only queries of type C# program and C# statements are supported"); } codeFile.Content = GetCode(result, codeFile.Query, @namespace); } private static IEnumerable<string> WrapInClass(IEnumerable<string> inputCode) { var s = new[] { "public class Program {" }; var e = new[] { "}" }; return s.Concat(inputCode).Concat(e); } private static IEnumerable<string> WrapInMain(IEnumerable<string> inputCode) { var s = new[] { "public static void Main() {" }; var e = new[] { "}" }; return s.Concat(inputCode).Concat(e); } private static IEnumerable<string> FilterTypes(IEnumerable<string> inputCode) { return inputCode.SkipWhile(l => l.Trim() != "// Define other methods and classes here"); } public static string GetCode(IEnumerable<string> content, Query query, string ns) { var code = string.Join(Environment.NewLine, content); var codeBuilder = new StringBuilder(); codeBuilder.AppendLine("using " + string.Join(";\r\nusing ", query.Namespaces.Union(StandardNamespaces)) + ";"); codeBuilder.AppendLine(string.Format("namespace {0} {{", ns)); codeBuilder.AppendLine(code); codeBuilder.AppendLine("}"); return codeBuilder.ToString(); } public static Assembly BuildAssembly(IEnumerable<CodeFile> codeFiles, Options options, string outputPath) { var providerOptions = new Dictionary<string, string> { { "CompilerVersion", "v4.0" } }; var provider = new CSharpCodeProvider(providerOptions); var assemblies = new[] { codeFiles.SelectMany(c => c.Query.GACReferences.Select(s => Assembly.Load(s).Location)), codeFiles.SelectMany(c => c.Query.RelativeReferences.Select(s => Assembly.LoadFrom(s).Location)), codeFiles.SelectMany(c => c.Query.OtherReferences), Assemblies }; var compilerparams = new CompilerParameters { GenerateExecutable = !string.IsNullOrWhiteSpace(options.StartupObject), OutputAssembly = options.IsInMemory ? null : outputPath, GenerateInMemory = true, IncludeDebugInformation = true, MainClass = options.StartupObject }; compilerparams.ReferencedAssemblies.AddRange(assemblies.SelectMany(a => a).ToArray()); var results = provider.CompileAssemblyFromSource(compilerparams, codeFiles.Select(f => f.Content).ToArray()); if (results.Errors.HasErrors) { var errors = new StringBuilder("Compiler Errors:\r\n"); foreach (CompilerError error in results.Errors) { errors.AppendFormat("File {0}, Line {1},{2}\t: {3}\r\n", error.FileName, error.Line, error.Column, error.ErrorText); } throw new Exception("Errors compiling:\r\n" + errors + "\r\n\r\n" + string.Join(Environment.NewLine, codeFiles.Select(f => f.FilePath + ":\r\n" + AddLineNumber(f.Content)))); } return results.CompiledAssembly; } private static string AddLineNumber(string code) { var lines = code.Split(new[] { Environment.NewLine }, StringSplitOptions.None); return string.Join(Environment.NewLine, lines.Select ((x, i) => (i + 1).ToString().PadLeft(4) + ": " + x)); } private static List<string> StandardNamespaces { get { return new List<string> { "System", "System.IO", "System.Text", "System.Text.RegularExpressions", "System.Diagnostics", "System.Threading", "System.Reflection", "System.Collections", "System.Collections.Generic", "System.Linq", "System.Linq.Expressions", "System.Data", "System.Data.SqlClient", "System.Data.Linq", "System.Data.Linq.SqlClient", "System.Xml", "System.Xml.Linq", "System.Xml.XPath" }; } } private static List<string> Assemblies { get { return new List<string> { "System.dll", "System.Core.dll", "System.Data.dll", "System.Xml.dll", "System.Xml.Linq.dll", "System.Data.Linq.dll", "System.Drawing.dll", "System.Data.DataSetExtensions.dll" }; } } } public class Query { public string Kind { get; set; } public List<string> Namespaces { get; set; } public List<string> GACReferences { get; set; } public List<string> RelativeReferences { get; set; } public List<string> OtherReferences { get; set; } } public enum CodeType { LinqStatements, // Wrap inside class and Main() method LinqProgramTypes, // Code after 'Define other methods and classes here' directly inside namespace LinqProgram, // Wrap all code inside class } public class CodeFile { public CodeType Type { get; set; } public string FilePath { get; set; } public string[] RawContent { get; set; } public Query Query { get; set; } public string Content { get; set; } } public class Options { private Options(string @namespace) { if (string.IsNullOrWhiteSpace(@namespace)) throw new ArgumentNullException("namespace"); this.CodeFiles = new List<CodeFile>(); this.Namespace = @namespace; } public static Options CreateInMemoryDll(string @namespace) { var options = new Options(@namespace); options.OutputFile = null; options.IsInMemory = true; options.StartupObject = null; return options; } public static Options CreateOnDiskDll(string @namespace, string outputFile) { if (string.IsNullOrWhiteSpace(outputFile)) throw new ArgumentNullException("outputFile"); var options = new Options(@namespace); options.OutputFile = outputFile; options.IsInMemory = false; options.StartupObject = null; return options; } public static Options CreateOnDiskExe(string @namespace, string outputFile, string startupObject) { if (string.IsNullOrWhiteSpace(outputFile)) throw new ArgumentNullException("outputFile"); if (string.IsNullOrWhiteSpace(startupObject)) throw new ArgumentNullException("startupObject"); var options = new Options(@namespace); options.OutputFile = outputFile; options.IsInMemory = false; options.StartupObject = startupObject; return options; } public Options AddCodeFile(CodeType type, string filePath) { this.CodeFiles.Add(new CodeFile { Type = type, FilePath = filePath }); return this; } public string Namespace { get; private set; } public string OutputFile { get; private set; } public List<CodeFile> CodeFiles { get; private set; } public string StartupObject { get; private set; } public bool IsInMemory { get; private set; } }

Tuesday, September 18, 2012

Using NuGet with LinqPad for free

Recently LinqPad got a great integration with NuGet in the paid version of the software

Unfortunately for the user of the free version it is a bit more difficult to use NuGet but it's not impossible.

Today I'm going to show you a way to use NuGet to get libraries into your LinqPad queries.

NuGet Package Explorer

You can download the NuGet Package Explorer from CodePlex

The primary use for this tool is to explorer, edit and publish NuGet packages. You could also use this tool to extract the files contained in the packages and put them in a folder, and that's what we'll do now.

First we need to Open a package from online feed

Here we could query for the package we want.

Select a package and click OK. We can now take a peek inside the package if we want. For us, we want to Export the content from the File menu.

You need choose a folder where to export the files of the package. Finally, in LinqPad we need to add a reference to the DLL by going in the Query Properties (F4) and clicking Browse. After that we are ready to start writing our queries.

Using the NuGet local cache folder

One another tip I've got for you is to add the local NuGet cache folder to the list of online feed as a shortcut for when we want to access a package we previously downloaded with the NuGet Package Explorer or even Visual Studio.

First, from the Tools menu select View NuGet download cache.

This will open a Windows Explorer on the folder %AppData%\Local\NuGet\Cache.

Now copy this path, go back to NuGet Package Explorer and Open a package from online feed. Then paste the path into Package source field and click Reload.

From now on you will be able to choose between the official NuGet.org feed and your local cache when you want to open NuGet packages.

NuGet.exe

Of course for the hardcore user of the command line interface one could use nuget.exe directly to download and extract the files rather than using the Package Explorer. I'm definitely not that hardcore. :-)

NuGet.exe is also available from the CodePlex site.

Friday, August 31, 2012

Using log4net MemoryAppender for unit testing

Sometimes unit testing existing code can be really hard. Especially when coding by exceptions (using Try.. Catch.. then do nothing!).

This will look a bit like this

From the outside we might not be able to test whether ReallyDoSomething has thrown an exception except if we can validate state changes done by ReallyDoSomething. And if we can't change the code much (like adding dependencies using parameters or change the return type of the method) we won't be able to write that test at all.

Another way is to use the logger to validate that.

The piece of code above use Log4Net as the logging framework and allow us to use it in our unit tests like this

[SetUp] public void SetUp() { this.memoryAppender = new MemoryAppender(); BasicConfigurator.Configure(this.memoryAppender); ... } [Test] public void TestSomething() { ... // Assert Assert.IsFalse( this.memoryAppender.GetEvents().Any(le => le.Level == Level.Error), "Did not expect any error messages in the logs"); }

Just like that we can intercept all the logged messages by our production code.

I've used this trick a few times now when other options were not available. Hope it helps.

Wednesday, August 15, 2012

TFS Queries : Searching in all work items

For an intro on LinqPad and the TFS API please read this post
For the list of all the posts in this series please read this one

Context

To search for work items effectively we can use a special TFS API on the WorkItemStore class. The query format is based the Work Item Query Language (or WIQL ) and we can't use Linq directly, but still, we can use LinqPad to write a quick little query.

Required references

Microsoft.TeamFoundation.Client
Microsoft.TeamFoundation.Common
Microsoft.TeamFoundation.VersionControl.Client
Microsoft.TeamFoundation.WorkItemTracking.Client

Query (C# Statements)

var tfs = TfsTeamProjectCollectionFactory
    .GetTeamProjectCollection(new Uri("http://localhost:8088/tfs"));

tfs.EnsureAuthenticated();

var workItemStore = tfs.GetService<WorkItemStore>();

var title = "phone";

var query = string.Format(@"
    Select [Id], [Work Item Type], [Title], [State]
    Where [Title] Contains '{0}'
    From WorkItems",
    title);

workItemStore.Query(query).Cast<WorkItem>()
    .Select(wi => new
    {
        wi.Id,
        wi.AreaPath,
        Type = wi.Type.Name,
        wi.Title,
        wi.State
    })
    .OrderBy(wi => wi.AreaPath).ThenBy(wi => wi.Type).ThenBy(wi => wi.Title)
    .Dump();

Result

Here we get the list of all the work items from all projects containing the word 'phone' in their title.

To find more about WIQL please take a look at the MSDN section on it.

Id	AreaPath	Type	Title	State
IOrderedEnumerable<> (1 item)
2	Project1	Bug	Fix the phone number on the Contact page	Done

Wednesday, August 1, 2012

TFS Queries: Listing all the branches

For an intro on LinqPad and the TFS API please read this post
For the list of all the posts in this series please read this one

Context

When it comes to branches, using the TFS Web Access or Team Explorer we can navigate around the source code explorer to see branches from various Team Project and related information. But to get the big picture there is an easier way using the TFS API.

Required references

Microsoft.TeamFoundation.Client
Microsoft.TeamFoundation.Common
Microsoft.TeamFoundation.VersionControl.Client

Query (C# Statements)

var tfs = TfsTeamProjectCollectionFactory.
    GetTeamProjectCollection(new Uri("http://localhost:8088/tfs"));

tfs.EnsureAuthenticated();

var versionControl = tfs.GetService<VersionControlServer>();

versionControl.QueryRootBranchObjects(RecursionType.Full)
    .Where(b => !b.Properties.RootItem.IsDeleted)
    .Select(s => new
    {
        Project = s.Properties.RootItem.Item
            .Substring(0, s.Properties.RootItem.Item.IndexOf('/', 2)),
        Properties = s.Properties,
        DateCreated = s.DateCreated,
        ChildBranches = s.ChildBranches
    })
    .Select(s => new 
    {
        s.Project,
        Branch = s.Properties.RootItem.Item.Replace(s.Project, ""),
        Parent = s.Properties.ParentBranch != null ?
            s.Properties.ParentBranch.Item.Replace(s.Project, "") : "",
        Version = (s.Properties.RootItem.Version as ChangesetVersionSpec)
            .ChangesetId,
        DateCreated = s.DateCreated,
        Owner = s.Properties.Owner,
        ChildBranches = s.ChildBranches
            .Where (cb => !cb.IsDeleted)
            .Select(cb => new
            {
                Branch = cb.Item.Replace(s.Project, ""),
                Version = (cb.Version as ChangesetVersionSpec).ChangesetId
            })
    })
    .OrderBy(s => s.Project).ThenByDescending(s => s.Version)
    .Dump();

Result

Here we can see a bit more information on the branches like related parent and child branches in a simple list.

IOrderedEnumerable<> (4 items)

Project

Branch

Parent

Version

DateCreated

Owner

ChildBranches

$/Project1

/Release v1.1.0.0

/Main

04/07/2012 9:18:01 PM

Domain\Pascal

IEnumerable<> (0 items)

$/Project1

/Main

28/06/2012 10:24:05 PM

Domain\Pascal

Branch	Version
IEnumerable<> (1 item)
/Release v1.1.0.0	14

$/Project2

/Team

/Main

04/07/2012 9:17:24 PM

Domain\Pascal

IEnumerable<> (0 items)

$/Project2

/Main

04/07/2012 9:17:02 PM

Domain\Pascal

Branch	Version
IEnumerable<> (1 item)
/Team	13

Wednesday, July 4, 2012

TFS Queries: Generating a changelog from the branch history

For an intro on LinqPad and the TFS API please read this post
For the list of all the posts in this series please read this one

Context

If you ever need to produce a changelog for a release you could do it by hand, looking at all the work items associated with the changesets or you could use the TFS API to generate it automatically.

Required references

Microsoft.TeamFoundation.Client
Microsoft.TeamFoundation.Common
Microsoft.TeamFoundation.VersionControl.Client
Microsoft.TeamFoundation.WorkItemTracking.Client

Query (C# Statements)

var branch = "$/Project1/Main";
var fromVersion = new ChangesetVersionSpec(1);
var toVersion = VersionSpec.Latest;

var tfs = TfsTeamProjectCollectionFactory
    .GetTeamProjectCollection(new Uri("http://localhost:8088/tfs"));

tfs.EnsureAuthenticated();

var versionControl = tfs.GetService<VersionControlServer>();

var history = versionControl.QueryHistory(branch, VersionSpec.Latest, 0, 
    RecursionType.Full, null, fromVersion, toVersion, int.MaxValue, false, 
    false, false);

history.OfType<Changeset>()
    .Select(x => new
    {
        x.ChangesetId,
        x.CreationDate,
        x.Committer,
        x.Comment,
        WorkItems = x.WorkItems.Select(wi => new
        {
            wi.Id,
            wi.Title,
            wi.Description,
            wi.State,
            wi.Reason
        })
    })
    .Dump("Branch history of: " + branch + 
        " from " + fromVersion.DisplayString + " to " + toVersion.DisplayString);
 
history.OfType<Changeset>()
    .OrderBy(x => x.ChangesetId)
    .SelectMany(x => x.WorkItems)
    .Select(wi => string.Format("[{0}] {1} #{2} - {3}", 
        wi.Reason, wi.Type.Name, wi.Id, wi.Title))
    .Dump("Changelog of branch: " + branch + 
        " from " + fromVersion.DisplayString + " to " + toVersion.DisplayString);

Result

With this we will get the list of all the changesets and associated work items from the history of the given branch.

Branch history of: $/Project1/Main from C1 to T

IEnumerable<> (4 items)

ChangesetId

CreationDate

Committer

Comment

WorkItems

28/06/2012 10:42:14 PM

Domain\Pascal

Change the phone number

Id	Title	Description	State	Reason
IEnumerable<> (1 item)
2	Fix the phone number on the Contact page		Done	Work finished

28/06/2012 10:39:58 PM

Domain\Pascal

Changed the version to 1.1.0.0

Id	Title	Description	State	Reason
IEnumerable<> (1 item)
1	Change the version number to 1.1.0.0		Done	Work finished

28/06/2012 10:31:55 PM

Domain\Pascal

Created a MVC4 Mobile application

IEnumerable<> (0 items)

28/06/2012 10:23:52 PM

Domain\Pascal

Added Main folder for the branch

IEnumerable<> (0 items)

Changelog of branch: $/Project1/Main from C1 to T

IEnumerable<String> (2 items)

[Work finished] Task #1 - Change the version number to 1.1.0.0

[Work finished] Bug #2 - Fix the phone number on the Contact page

Wednesday, June 20, 2012

NuGet integration in LinqPad

NuGet is amazing in Visual Studio. With it I can easily download and add references to popular libraries in my project with only a few clicks in Manage NuGet Packages or keystrokes when using the Package Manager Console. I really like that because I can try out new libraries without too much effort. I also like LinqPad to try out snippets of code. But if I want to try a library I have to download the binaries from a web site, unzip them on my disk and finally add references to the assemblies from my query before I can start to play around with it.

The new version of LinqPad (v4.42.04 currently still in beta) finally gets an integration with NuGet directly in the tool but only with the Premium Edition. Let's take a look at how it works.

In the Query Properties (F4) we have a new option: Add NuGet...

This will open the NuGet Manager window

On the left we can see all the NuGet packages we already downloaded. Those will be stored in our local AppData folder and be available for all queries. In the middle section we have the NuGet feed search where we can browse for packages to download. Finally in the section on the right we see all the information on the currently selected package.

To add a package to the current query simply click Add To Query. If the package was not already downloaded LinqPad will do it now and after that it will appear in the left section. Finally we can select namespaces to use in the query by clicking Add namespace.

LinqPad will also automatically check for updates for us so we just have to click Update next to the package. All the queries using this package will get updated to the latest version too.

We are not limited to the official NuGet feed but we could also add new feeds to connect to, even our own if we wish. We can add new feeds by clicking the Settings button in the lower left of the window.

Here we can add a remote feed or a local folder if we wish. After that we'll have the option to choose which feed to use in the main NuGet Manager window using the list down the middle section.

Going back to the Query Properties window all the packages we added will be shown as a reference with the location set to NuGet. If we click on NuGet we can see the current version of the package and an option to Show assemblies in this package.

Here we can exclude some assemblies from the package if we need to.

One additional FREE new feature this time we get in this version is the ability to browse for namespaces to add from existing references. In the Additional Namespace Imports tab click on Pick from assemblies

We can now select one assembly to see the list of available namespaces from it. Then simply select the namespaces we want and click Add Selected Namespaces.

Summary

I know that not everyone have the Premium version of LinqPad but still I wanted to show you what appends when 2 amazing productivity tools collide! I guess it is possible to use some NuGet Powershell cmdlet to do part of the job. It think a stand alone NuGet client could be a good idea for someone to come up with too.

Anyway, hope you get the chance to play with those two together