Productive Rage

29 February 2012

Being a dirty GitHub-to-BitBucket turncoat

After having a bit of a run-in with GitHub a few weeks ago (GitHub and I might be having a falling-out) I was wondering if there was any easy migration route from GitHub over to BitBucket, which I'd been using to house a prototype project I've been doing related to work. We use Mercurial (with Kiln) and after getting over the initial CVS-to-Mercurial teething problems I'm well into it (though being better than CVS isn't that huge an undertaking! :)

The last straw was when I wanted to rename a project within a Visual Studio solution. Using TortoiseHg this is easy; perform the rename, use some "right-click / TortoiseHg / Guess renames" action. (Command line options also available!) But with Git I was struggling to find the best way to do it - I read a load of articles and a handful of StackOverflow posts. It seems like renaming files without changing their contents and then changing their contents will ensure the history is maintained. Usually. Except some times. Or during a full moon.

Ok, I admit, I didn't actually bother trying! And at this point I think I'm glad I didn't, since moving over to BitBucket turned out to be really easy with built-in tools.

A false start

After doing some initial reading around I found out about the BitBucket "Import Repository" functionality. Amazing! Point it at my GitHub repositories and import them into new BitBucket Mercurial repos - ideal! So I select "Git/GitHub" as the source and it set "Git" as the "Repository type", fair enough - it's realised that it has to import from Git! Selected Language as "C#" and hit "Import" and.. it all looked rosy!

I set up Username Aliases to my BitBucket user and had a poke around the history and found it all looked good. It even imported the tags from GitHub which I'd been concerned about since one of the articles I'd read had put some doubt in my mind regarding those.

Then I tried to clone the repository onto my computer. HTTP 501 "Not Implemented". Hmmm.. oh well, it was late, maybe I was doing something dumb.

So then I tried again the next day at work. Same issue. Not too reassuring :( I had a poke around the "My repositories" section and saw that the import repo was marked as being "Git". I tried cloning it and it worked. So imported yes, converted to Mercurial no. Back to the drawing board.

Hg Convert

Thankfully the solution I've ended up with wasn't much more complicated and uses a standard Mercurial Extension (Convert). The quick-fire version is to:

1. Ensure the "convert" extension is enabled by going into Global Settings in TortoiseHg (or adding "convert=" to the "[extensions]" section of your mercurial.ini file manually if you're Tortoise-adverse)
2. Open a command prompt
3. Enter "hg convert c:\folder\name\of\git\repo" (quoting the location if there are any spaces)
4. This will create a new folder in the command prompt's current location, taking the Git repo's name and suffixing with "-hg" (so "repo-hg" in this example)
5. Move into folder and enter "hg update" to pull in the files
6. Profit! :)

I created a new (Mercurial!) repository in BitBucket and pushed from the local repo up to it. Easy!

The repository I was playing with here had various commits by me under different usernames. I set up aliases to my BitBucket account for these but the convert extension offers options to remap author names so that these aliases aren't required (see the Convert docs).

Looking back

I must admit that part of the reason I started up the GitHub account was to build up a portfolio of demo code for when the day comes that I decide I want to look for alternative employment. And for some reason having a GitHub account just sounds cooler than BitBucket! But maybe that's just been engrained into me by the many many references to a GitHub portfolio that I've heard made on Hacker News over the years. Plus I'm going to miss the GitHub cat logo - BitBucket just doesn't seem quite as friendly without it!

Posted at 20:44

Comments

26 February 2012

The artist previously known as the AutoMapper-By-Constructor

I've had a series of posts that was initiated by a desire to integrate AutoMapper more easily with classes that are instantiated with so-called "verbose constructors"..

• Teaching AutoMapper about "verbose constructors"
• AutoMapper-By-Constructor without AutoMapper.. and faster
• Extendable LINQ-compilable Mappers
• The Less-Effort Extendable LINQ-compilable Mappers

.. that ended up going on somewhat of a tangent and enabled the generation of compilable converters (using LINQ Expressions) that didn't utilise AutoMapper for the majority of simple cases.

While the original intention of the project was to handle the conversion to these "verbose constructor"-based types, it struck me a few days ago that it shouldn't be much work to put together a class similar to the CompilableTypeConverterByConstructor that instead instantiates a type with a parameter-less constructor and sets the data through property-setters rather than by converter. The concept that started this all off in my head was a service that exposed xml-serialisable objects at the boundary but used "always-valid" internal representations (ie. immutable data where all values were specified and validated by constructor) - I wanted a way to convert to internal types. But with this property-setting approach the code could transform both ways.

(Just a quick side-node that for transformations to data-set-by-property types, AutoMapper is actually a much more full-featured package but for what I had in mind the simple name-matching in my project coupled with the significantly improved performance from the compiled converters was a better fit).

I still find LINQ Expressions hard to write

I envisaged something along the lines of a new class

public class CompilableTypeConverterByPropertySetting<TSource, TDest>
    : ICompilableTypeConverter<TSource, TDest> where TDest : new()
{
    public CompilableTypeConverterByPropertySetting(
        IEnumerable<ICompilablePropertyGetter> propertyGetters,
        IEnumerable<PropertyInfo> propertiesToSet)
    {
        // Do constructor work..

where the number of propertyGetters would match the number of propertiesToSet. I won't go back over the ICompilableTypeConverter since it's not that important right this second but the property getters are:

public interface ICompilablePropertyGetter : IPropertyGetter
{
    /// <summary>
    /// This must return a Linq Expression that retrieves the value from SrcType.Property as
    /// TargetType - the specified "param" Expression must have a type that is assignable to
    /// SrcType.
    /// </summary>
    Expression GetPropertyGetterExpression(Expression param);
}

public interface IPropertyGetter
{
    /// <summary>
    /// The type whose property is being accessed
    /// </summary>
    Type SrcType { get; }

    /// <summary>
    /// The property on the source type whose value is to be retrieved
    /// </summary>
    PropertyInfo Property { get; }

    /// <summary>
    /// The type that the property value should be converted to and returned as
    /// </summary>
    Type TargetType { get; }

    /// <summary>
    /// Try to retrieve the value of the specified Property from the specified object (which
    /// must be of type SrcType) - this will throw an exception for null or if retrieval fails
    /// </summary>
    object GetValue(object src);
}

So this should be easy! All I need is to create LINQ Expressions that can take a ParameterExpression of type TSource, use it to instantiate a new TDest and set each of the properties that I already have. And I've already got Expressions to retrieve the data from the TSource instance for each of the properties!

private Func<TSource, TDest> GenerateCompiledConverter()
{
    // Declare an expression to represent the src parameter
    var src = Expression.Parameter(typeof(TSource), "src");

    // Declare a local variable that will be used within the Expression block to have a new
    // instance assigned to it and properties set
    var dest = Expression.Parameter(typeof(TDest));

    // Build up a list of Expressions that:
    // 1. Instantiate a new TDest instance
    var newInstanceGenerationExpressions = new List<Expression>
    {
        Expression.Assign(
            dest,
            Expression.New(typeof(TDest).GetConstructor(new Type[0]))
        )
    };

    // 2 Set properties on the new instance
    for (var index = 0; index < _propertiesToSet.Count; index++)
    {
        newInstanceGenerationExpressions.Add(
            Expression.Call(
                dest,
                _propertiesToSet[index].GetSetMethod(),
                _propertyGetters[index].GetPropertyGetterExpression(src)
            )
        );
    }

    // 3. Return the reference
    newInstanceGenerationExpressions.Add(
        dest
    );

    // Return compiled expression that instantiates a new object by retrieving properties
    // from the source and passing as constructor arguments
    return Expression.Lambda<Func<TSource, TDest>>(
        Expression.Block(
            new[] { dest },
            newInstanceGenerationExpressions
        ),
        src
    ).Compile();
}

(Take it as read that _propertiesToSet and _propertyGetters are PropertyInfo[] and ICompilablePropertyGetter[] that are validated and set as class-scoped members by the constructor).

And indeed it does look easy! And I'm kinda wondering what all the fuss was about, but it took me a fair bit of tinkering and reasoning to get here since the LINQ Expression tutorials and examples just aren't that easy to track down! And it's not like you can easily take apart arbitrary example code like when dealing with IL (see the IL Disassembler mention in Dynamically applying interfaces to objects).

But I got there in the end! The only slightly odd thing is that the last expression has to be the ParameterExpression "dest" that we've constructed, otherwise the block won't return anything - it just returns the result of the last expression.

Ok. I've actually lied. That isn't quite all of it. As an ICompilableTypeConverter, the CompilableTypeConverterByPropertySetting should be able to handle null values so that the CompilableTypeConverterPropertyGetter class can take any ICompilableTypeConverter reference and use it to retrieve and convert property values.. even when they're null. So the last section becomes:

    // Return compiled expression that instantiates a new object by retrieving properties
    // from the source and passing as constructor arguments
    return Expression.Lambda<Func<TSource, TDest>>(

        Expression.Condition
            Expression.Equal(
                src,
                Expression.Constant(null)
            ),
            Expression.Constant(default(TDest), typeof(TDest)),
            Expression.Block(
                new[] { dest },
                newInstanceGenerationExpressions
            )
        ),

        src

    ).Compile();

.. so that it will return the default value to TDest (null unless TDest is a ValueType) if the TSource value is null.

Wrapping in a Factory

As with the similar CompilableTypeConverterByConstructor class there's a factory class which will examine given TSource and TDest types and try to generate a CompilableTypeConverterByPropertySetting<TSource, TDest> instance based on the ICompilablePropertyGetter set it has (and the INameMatcher for matching source and destination properties).

I've also updated the ExtendableCompilableTypeConverterFactory (see The Less-Effort Extendable LINQ-compilable Mappers) such that it is more generic and doesn't insist on being based around CompilableTypeConverterByConstructorFactory. There is now a static helper class to instantiate an ExtendableCompilableTypeConverterFactory instance based upon whether the target type is to have its data set by-constructor or by-property-setting since the changes to ExtendableCompilableTypeConverterFactory have made it very abstract!

Splitting the AutoMapper dependency

Since the majority of work in this solution no longer requires AutoMapper, I've broken out a separate project "AutoMapperIntegration" which houses the AutoMapperEnabledPropertyGetter and AutoMapperEnabledPropertyGetterFactory classes so now the main project has no AutoMapper reference. My original intention was improve how AutoMapper worked with by-constructor conversions and this functionality is still available - without taking advantage of the compiled converters - by referencing the main project along with AutoMapperIntegration (and so the example in Teaching AutoMapper about "verbose constructors" is still applicable).

And so I've renamed the solution itself to...

The Compilable Type Converter!

Yeah, yeah, not too imaginative a title, I will admit! :)

I've actually moved my code over to BitBucket (see upcoming post!) from GitHub, so the code that I've been talking about can now be found at:

https://bitbucket.org/DanRoberts/compilabletypeconverter

An apology

This has been a particularly dry and largely self-involved post but if the Compilable Type Converter sounds like it might be useful to you, check out that BitBucket link and there's an introduction on the Overview page which jumps straight into example code.

Examples

To demonstrate the generation of a converter from a generic SourceType class to one that is based upon verbose constructors:

// Prepare a converter factory using the base types (AssignableType and
// EnumConversion property getter factories)
var nameMatcher = new CaseInsensitiveSkipUnderscoreNameMatcher();
var converterFactory = ExtendableCompilableTypeConverterFactoryHelpers.GenerateConstructorBasedFactory(
    nameMatcher,
    new ArgsLengthTypeConverterPrioritiserFactory(),
    new ICompilablePropertyGetterFactory[]
    {
        new CompilableAssignableTypesPropertyGetterFactory(nameMatcher),
        new CompilableEnumConversionPropertyGetterFactory(nameMatcher)
    }
);

// Extend the converter to handle SourceType.Sub1 to ConstructorDestType.Sub1 and
// IEnumerable<SourceType.Sub1> to IEnumerable<ConstructorDestType.Sub1>
// - This will raise an exception if unable to create the mapping
converterFactory = converterFactory.CreateMap<SourceType.Sub1, ConstructorDestType.Sub1>();

// This will enable the creation of a converter for SourceType to ConstructorDestType
// - This will return null if unable to generate an appropriate converter
var converter = converterFactory.Get<SourceType, ConstructorDestType>();
if (converter == null)
    throw new Exception("Unable to obtain a converter");

var result = converter.Convert(new SourceType()
{
    Value = new SourceType.Sub1() { Name = "Bo1" },
    ValueList = new[]
    {
        new SourceType.Sub1() { Name = "Bo2" },
        null,
        new SourceType.Sub1() { Name = "Bo3" }
    },
    ValueEnum = SourceType.Sub2.EnumValue2
});

public class SourceType
{
    public Sub1 Value { get; set; }
    public IEnumerable<Sub1> ValueList { get; set; }
    public Sub2 ValueEnum { get; set; }
    public class Sub1
    {
        public string Name { get; set; }
    }
    public enum Sub2
    {
        EnumValue1,
        EnumValue2,
        EnumValue3,
        EnumValue4,
        EnumValue5,
        EnumValue6,
        EnumValue7,
        EnumValue8
    }
}

public class ConstructorDestType
{
    public ConstructorDestType(Sub1 value, IEnumerable<Sub1> valueList, Sub2 valueEnum)
    {
        if (value == null)
            throw new ArgumentNullException("value");
        if (valueList == null)
            throw new ArgumentNullException("valueList");
        if (!Enum.IsDefined(typeof(Sub2), valueEnum))
            throw new ArgumentOutOfRangeException("valueEnum");
        Value = value;
        ValueList = valueList;
        ValueEnum = valueEnum;
    }
    public Sub1 Value { get; private set; }
    public IEnumerable<Sub1> ValueList { get; private set; }
    public Sub2 ValueEnum { get; private set; }
    public class Sub1
    {
        public Sub1(string name)
        {
            name = (name ?? "").Trim();
            if (name == "")
                throw new ArgumentException("Null/empty name specified");
            Name = name;
        }
        public string Name { get; private set; }
    }
    public enum Sub2 : uint
    {
        EnumValue1 = 99,
        EnumValue2 = 100,
        EnumValue3 = 101,
        EnumValue4 = 102,
        EnumValue5 = 103,
        enumValue_6 = 104,
        EnumValue7 = 105
    }
}

.. and the equivalent where the destination types are based upon property-setting:

// Prepare a converter factory using the base types (AssignableType and EnumConversion property
// getter factories)
var nameMatcher = new CaseInsensitiveSkipUnderscoreNameMatcher();
var converterFactory = ExtendableCompilableTypeConverterFactoryHelpers.GeneratePropertySetterBasedFactory(
    nameMatcher,
    CompilableTypeConverterByPropertySettingFactory.PropertySettingTypeOptions.MatchAsManyAsPossible,
    new ICompilablePropertyGetterFactory[]
    {
        new CompilableAssignableTypesPropertyGetterFactory(nameMatcher),
        new CompilableEnumConversionPropertyGetterFactory(nameMatcher)
    }
);

// Extend the converter to handle SourceType.Sub1 to ConstructorDestType.Sub1 and
// IEnumerable<SourceType.Sub1> to IEnumerable<ConstructorDestType.Sub1>
// - This will raise an exception if unable to create the mapping
converterFactory = converterFactory.CreateMap<SourceType.Sub1, PropertySettingDestType.Sub1>();

// This will enable the creation of a converter for SourceType to ConstructorDestType
// - This will return null if unable to generate an appropriate converter
var converter = converterFactory.Get<SourceType, PropertySettingDestType>();
if (converter == null)
    throw new Exception("Unable to obtain a converter");

var result = converter.Convert(new SourceType()
{
    Value = new SourceType.Sub1() { Name = "Bo1" },
    ValueList = new[]
    {
        new SourceType.Sub1() { Name = "Bo2" },
        null,
        new SourceType.Sub1() { Name = "Bo3" }
    },
    ValueEnum = SourceType.Sub2.EnumValue2
});

public class SourceType
{
    public Sub1 Value { get; set; }
    public IEnumerable<Sub1> ValueList { get; set; }
    public Sub2 ValueEnum { get; set; }
    public class Sub1
    {
        public string Name { get; set; }
    }
    public enum Sub2
    {
        EnumValue1,
        EnumValue2,
        EnumValue3,
        EnumValue4,
        EnumValue5,
        EnumValue6,
        EnumValue7,
        EnumValue8
    }
}

public class PropertySettingDestType
{
    public Sub1 Value { get; set; }
    public IEnumerable<Sub1> ValueList { get; set; }
    public Sub2 ValueEnum { get; set; }
    public class Sub1
    {
        public string Name { get; set; }
    }
    public enum Sub2 : uint
    {
        EnumValue1 = 99,
        EnumValue2 = 100,
        EnumValue3 = 101,
        EnumValue4 = 102,
        EnumValue5 = 103,
        enumValue_6 = 104,
        EnumValue7 = 105
    }
}

Posted at 21:39

Comments

13 February 2012

GitHub and I might be having a falling-out

Last week I encountered some problems with pushing and pulling against my GitHub repositories. I've been using them (only on and off, granted) for a good few months now, maybe a year, and I've had no real issues with it. In fact for a free service it's been great - I like the UI (and the logos! :) and everything has been straight forward and let me play with Git for the first time.

But then it stopped playing with me.

Clicking on the "Help" link when in GitHub goes to help.github.com which has a "Site Status" indicator, which is cool. It was showing a happy green "All systems operational" display.

I thought maybe it was TortoiseGit since it had been bugging me about updates for a while. So I uninstalled it and re-installed it and went through some questions that I remembered not really understanding from last time.. :( One tutorial I'd read those months ago had recommended using OpenSSH while another recommended PuTTY. I couldn't remember which I'd used in the past. I'd stored an SSH password somewhere but couldn't remember what it was for. So I tried sticking with the defaults (always a reasonable first-approach plan! :) and still wasn't getting it to communicate with GitHub properly..

Turns out that now the status page was showing that the service was indeed encountering difficulties. And I see now they have a Twitter account to keep you abreast of such issues. That's a nice touch.

So I waited it out and then tried to get TortoiseGit working again. I can't remember exactly what hoops I had to jump through in the end. There was a load of faff about re-generating SSH public keys and registering them in the Admin section of my GitHub account. And some more and back and forth where authentication attempts would keep getting rejected because some PuTTy-related component insisted on being run manually (although this was somehow ironed out too such that it would fire up automatically at the first attempt).

And then I finally got it going again and couldn't successfully push a new tag up to GitHub. Then I remembered you have to choose "Push tags" in Git Sync. It won't do it otherwise.

I think, on the whole, I've just been spoilt by all my recent dealing with Mercurial. I work solely with Windows and I'd sort of heard vague mutterings about Mercurial being an easier fit with Windows than Git and I think it's taken until now to get some first-hand experience.

At the end of the day

I'm not sure how I feel about this whole episode. Don't get me wrong, I think GitHub's handling was basically first-rate - they openly admit when issues are detected and then keep you up-to-date with progress on fixes. That's awesome. It's a bit annoying that they hadn't noticed the problem when I first encountered it which is why, I presume, they were reporting "All systems operational" when all systems were not entirely operational. But I'm really not annoyed with them, I'm annoyed with how much harder work TortoiseGit with Windows is compared to TortoiseHg. Maybe my DVCS GUI reliance is my problem! For a lot of things, I love the command line - but I feel much more comfortable with a GUI for source control; so I can easily see what's changed, added, deleted, so I can do a last-minute diff on every file before I commit - it just seems easier!

I think I'll hold-fire on any hasty decisions for now. In large part because I've been mostly happy with the experience so far and in part because seeing the GitHub logos every time I check in online entertains me! :D

Posted at 22:16

Comments