Blog

A catalogue of my discoveries in software development and related subjects, that I think might be of use or interest to everyone else, or to me when I forget what I did!

Developer Tools

June 17, 2019

I try to maintain a toolkit of useful apps for doing my daily development tasks. Some of these I use very frequently, others not so much but they are useful to know about. I thought I'd catalogue them on my blog so that I remember them when I'm setting up a new machine :)
Tool Name Description
Microsoft Visual Studio I think this one goes without saying, but if anyone getting into development needs to choose an IDE I'd highly recommend starting here! It pretty much does everything you need (solutions, projects, code editing, compiling, debugging, NuGet package management, profiling, source control and more) and at the time of writing is available for Windows and Mac. The main competitor being JetBrains Rider which is fully cross platform and includes Re-Sharper refactorings, but as of yet has not tempted me away from the staple of Visual Studio. There are free editions of Visual Studio suitable for most people.
JetBrains Re-Sharper A plugin for Visual Studio which has many extensions and helpers to refactor your code, spot potential issues, decompile .NET assemblies, performance tracing etc. It does have a cost associated with it and I don't always install it as I don't like the idea of being dependent on it and there is a lot of cross over in functionality provided by Visual Studio itself of other free 3rd party tools. However more and more I am liking a lot of the features and it becoming a staple in my day to day developments.
CodeMaid A free plugin for Visual Studio which provides shortcuts for cleaning up code files, such as ensuring the order of code within classes, removing and sorting "using" statements etc. You can also download my preferred settings for CodeMaid.
NCrunch A plugin for Visual Studio which provides test code coverage and a automatic background test runner to keep you well informed of uncovered lines or broken tests while you develop. This one also has a cost associated with it but I'm yet to find anything in the free software space that comes close to the functionality.
Notepad++ A free cross platform text editor which is well maintained and comes with a lot of features for working with text files. It's not a "code editor", as such although it supports syntax highlighting, but it's useful for quickly viewing or editing all kinds of text files.
VS Code A free cross platform extensible IDE/text editor by Microsoft. For me, this is the middle ground between opening Notepad++ and opening Visual Studio. I also like to use VS Code when working on any front-end projects such as those built using Webpack due to the lack of Visual Studio project files in those kind of projects and because of the built in terminal window.
Sourcetree A free GUI for Git. One of the best I've tried and adds real value vs using the Visual Studio plugin or going fully command line.
Fiddler A free tool to aid debugging web based application. It can capture web traffic as well as reply packets, intercept calls and more.
Wireshark A free tool to aid debugging network traffic. Generally I use this when Fiddler can't intercept the traffic and I need something a little further down the network stack for capturing traffic.
ILSpy A free tool for decompiling .NET assembles.
Multi Commander A free dual pane file explorer tool with many extensions and helpful functions for dealing with different types of file. Most of the time I find Windows Explorer fine, but sometimes an alternative tool with more options can be useful. From all the ones I tried this is currently my favourite.
FAR - Find and Replace A free tool for performing 2 useful operations - 1 is replace names within files (multi rename) and 2 is replace text within files. This is useful when you want to create a new project based on another and want to quickly rename all project files and swap out the namespaces in all code files.
mRemoteNG A free tool for managing connections to remote machines including RDP, SSH and Web interfaces.
WinMerge A free tool for comparing and merging files and folders.
Permalink: Developer Tools

Visual Studio 2017/2019 Not Remembering Custom Fonts and Colours

April 04, 2019

I've had issues with both VS2017 and now VS2019 where applying my custom fonts/colour scheme is not maintained between sessions. The same trick worked in VS2019 as what I discovered in VS2017, so this time I'm blogging it! Basically, import your custom colour scheme as usual using the "Import and Export Settings" wizard. Now go to Tools > Options > General and switch the "Color Theme" to any other theme than the current one. Now switch the theme back. That's it! For some reason this seems to persist your customisation of the theme whereas without switching themes the changes get lost.
Permalink: Visual Studio 2017/2019 Not Remembering Custom Fonts and Colours

Testing if XML has deserialized correctly

March 24, 2019

XML is pretty old tech and without a schema is a bit of a pain to work with! A semi saving grace is using Visual Studio's "Paste XML as Classes" option (Paste Special) which will generate C# classes capable of representing the XML you had on the clipboard (using the XmlSerializer). However the caveat to this is that it only generates code for the exact xml you have used, so any optional attributes/elements or collections that only have 1 item in them will be generated incorrectly and will silently start dropping information when you deserialize another file with slightly different xml content. To combat this, I wrote a simple XmlSchemaChecker class which takes the content of an XML file and it's deserialized equivalent and ensures that every piece of data from the file is represented within the instance. It logs these problems when running with Debug logging enabled and is called from the class responsible for deserializing files.
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Xml;
using System.Xml.Serialization;
using Microsoft.Extensions.Logging;

namespace Deserialization
{
    public class XmlSchemaChecker : IXmlSchemaChecker
    {
        private readonly ILogger<XmlSchemaChecker> _logger;

        public XmlSchemaChecker(ILogger<XmlSchemaChecker> logger)
        {
            _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }

        public void LogSchemaWarnings<T>(string originalXmlFilePath, T deserialized)
        {
            if (!_logger.IsEnabled(LogLevel.Debug)) return;

            var originalXml = File.ReadAllText(originalXmlFilePath);
            var newXml = ReSerialize(deserialized);

            var originalValues = GetXmlValues(originalXml);
            var newValues = GetXmlValues(newXml);

            var missingItems = originalValues.Except(newValues).ToList();

            if (missingItems.Any())
            {
                _logger.LogDebug("Schema for {filename} was not fully deserialized. Missing items: {missingItems}", originalXmlFilePath, missingItems);
            }
        }

        private static void ProcessNodes(ISet<string> values, Stack<string> paths, IEnumerable nodes)
        {
            foreach (var node in nodes)
            {
                switch (node)
                {
                    case XmlDeclaration _:
                        continue;
                    case XmlElement element:
                        {
                            paths.Push(element.Name);

                            foreach (var att in element.Attributes)
                            {
                                if (att is XmlAttribute xmlAttribute && xmlAttribute.Name != "xmlns:xsd" && xmlAttribute.Name != "xmlns:xsi")
                                {
                                    values.Add($"{string.Join(":", paths.Reverse())}:{xmlAttribute.Name}:{CleanseValue(xmlAttribute.Value)}");
                                }
                            }

                            if (element.HasChildNodes)
                            {
                                ProcessNodes(values, paths, element.ChildNodes);
                            }

                            paths.Pop();
                            break;
                        }
                    case XmlText text:
                        {
                            values.Add($"{string.Join(":", paths.Reverse())}:{text.ParentNode.Name}:{CleanseValue(text.InnerText)}");
                            break;
                        }
                }
            }
        }

        private static string CleanseValue(string value)
        {
            return value.Replace("\r\n", "\n").Replace("\t", "").Trim(' ', '\n');
        }

        private static IEnumerable<string> GetXmlValues(string xml)
        {
            var values = new HashSet<string>();
            var paths = new Stack<string>();
            var doc = new XmlDocument();
            doc.LoadXml(xml);

            ProcessNodes(values, paths, doc.ChildNodes);

            return values;
        }

        private static string ReSerialize<T>(T item)
        {
            var xmlSerializer = new XmlSerializer(typeof(T));
            var output = new System.Text.StringBuilder();

            using (var outputStream = new StringWriter(output))
            {
                xmlSerializer.Serialize(outputStream, item);
            }

            return output.ToString();
        }
    }
}
Permalink: Testing if XML has deserialized correctly

.NET Core Configuration Wire-Up

March 04, 2019

In .NET Core the way you wire you your configuration classes has changed since .NET Framework. Typically in netfx I would define interfaces in my application code and then in the composition root (such as a web site) I would create classes which implement these and wrap the ConfigurationManager. I like that approach because it's easy to switch out the implementation later for specific classes, such as using configuration DB or even having some custom calculations or parsing driving the configuration. In netcore, it seems Microsoft are pushing you down the route of POCO classes for configuration. You still have the option to create interfaces on top of these classes for your downstream consumers, or to pass these in as classes directly, or wrap the dependency in an IOptions interface. The general approach I see online is to create entries in ConfigureServices within the Startup.cs which uses "Configuration.Bind" to hydrate these classes, but this gets quite messy in my opinion, as it creates 3 lines of code inside ConfigureServices per configuration object and leaks concerns of implementation into the Startup.cs, away from the implementation class itself. e.g.
// startup.cs - ConfigureServices
var someSettings = new SomeSettings();
Configuration.Bind("SomeSettings", someSettings);
services.AddSingleton<ISomeSettings>(someSettings);

// SomeSettings.cs
public class SomeSettings : ISomeSettings
{
    public int SomeIntSetting { get; set; }
}
My preferred approach is to straddle old and new.. I will create an implementation in the composition root which "wraps" the application configuration, but instead of this being the old "ConfigurationManager" it simply takes a dependency on IConfiguration, that way if you want to deviate from this you only change the class that you are intending to change and also it keeps the details of how those configurations are materialised to the class that defines them. e.g.
// startup.cs - ConfigureServices
services.AddSingleton<ISomeSettings, SomeSettings>();

// SomeSettings.cs
public class SomeSettings : ISomeSettings
{
    public SomeSettings(IConfiguration configuration)
    {
        configuration.Bind("SomeSettings", this);
    }

    public int SomeIntSetting { get; set; }
}
Permalink: .NET Core Configuration Wire-Up

Exposing Kafka from Rancher/K8S VM to Local Machine

March 01, 2019

Following on from my previous post on setting up Rancher/K8S on RancherOS in a VM in Windows for local development, a common task will be setting up container services within the cluster but then accessing those services from your local Windows machine (e.g. while developing in Visual Studio). In a lot of cases this is probably straightforward, either exposing ports directly using a service or using Ingress to route host headers to the correct internal service. However in the case of Kafka it's a bit more complex due to the way in which the brokers address themselves when the initial connection is received and the broker list is sent back. In a nutshell, the default Kafka setup from the Catalog Apps in Rancher binds the brokers to their POD IP, when the broker list is sent to Windows it cannot address these IPs (unless you want to set up some kind of natting). After some Googling and help from the following posts: https://rmoff.net/2018/08/02/kafka-listeners-explained/ https://github.com/helm/charts/issues/6670 I came up with the following instructions: STEP 1 (install Kafka in cluster): Install Kafka from the Rancher catalogue
  1. your-dev-cluster > default > Catalog Apps > Launch
  2. find and select "Kafka"
  3. switch off the "Topics UI Layer 7 Loadbalancer" (near the bottom) - don't need it in dev.
  4. click "Launch"
  5. .. Wait until all the kafka services are running ..
  6. You can now verify that the Landoop UI is running and seeing brokers by visiting the endpoint is has produced, e.g. http://rancherdev.yourdomain:30188 <-- random port, check what it says!!
Kafka is now available in the cluster, but not from Windows. Continue with step 2 --> STEP 2 (expose Kafka externally): Change the Kafka startup command for multiport listening
  1. your-dev-cluster > default > workloads > kafka-kafka
  2. Three dots, click "Edit"
  3. Click "show advanced options"
  4. Under Command > Entrypoint - paste the following:
    sh -exc 'export KAFKA_BROKER_ID=${HOSTNAME##*-} && \export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_IP}:9092,EXT://rancherdev.yourdomain.com:$((9093 + ${KAFKA_BROKER_ID})) && \export KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXT:PLAINTEXT && \export KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT && \exec /etc/confluent/docker/run'
  5. Click "Upgrade"
Add service discovery for the new ports
  1. your-dev-cluster > default > Service Discovery
  2. Click "View/Edit YAML" on kafka-kafka..
  3. Use the following lines for section "spec > ports" (assuming you have 3 instances of Kafka)
    ports:
      - name: broker
        port: 9092
        protocol: TCP
        targetPort: 9092
      - name: broker-ext0
        port: 9093
        protocol: TCP
        targetPort: 9093
      - name: broker-ext1
        port: 9094
        protocol: TCP
        targetPort: 9094
      - name: broker-ext2
        port: 9095
        protocol: TCP
        targetPort: 9095
    
Configure nginx to use TCP ConfigMap
  1. your-dev-cluster > system > workloads > nginx-ingress-controller
  2. Three dots > edit
  3. Environment variables:
  4. "Add from Source" > "Config Map" > "tcp-services"
  5. Click "Upgrade"
Expose the port using Ingress TCP ConfigMap
  1. your-dev-cluster > system > resources > config maps > ns: ingress-nginx > tcp-services
  2. Three dots, click "Edit"
  3. Add the following entries:
            - key = 9093
            - value = kafka/kafka-kafka:9093
            - key = 9094
            - value = kafka/kafka-kafka:9094
            - key = 9095
            - value = kafka/kafka-kafka:9095
    
Reboot the kafka services
  1. your-dev-cluster > default > workloads > tick all and click 'redeploy'
Now from Windows try telnet to rancherdev.yourdomain.com 9093/9094/9095 or even better from WSL bash, install kafkacat and run: kafkacat -b rancherdev.yourdomain.com:9093 -L
Permalink: Exposing Kafka from Rancher/K8S VM to Local Machine

Setting up a Kubernetes cluster using Rancher on RancherOS

February 17, 2019

Little cheat sheet for setting up a single node Kubernetes/Rancher on a developer machine using Hyper-V without tying it to the DHCP IP address that was issued at the time of creation. Setup Rancher on RancherOS
  1. Download the RancherOS Hyper-V ISO image from the GitHub repo
  2. Setup a Hyper-V VM with the bootable ISO set as the boot device (with Internet connectivity - I used 4 vCPU, 16GB RAM and 500GB vHDD)
  3. Boot the VM and allow Linux to boot
  4. Type the following command (uses a password to avoid SSH keys):
    sudo ros install -d /dev/sda --append "rancher.password=yourpassword"
    
  5. Reboot and skip the CD boot step (i.e. boot from the hard disk)
  6. Login with "rancher" and "yourpassword" - at this point you may wish to get the IP and switch to another SSH client such as PuTTY and login from there.
  7. Create an SSL certificate for your "rancherdev" domain - from your rancher home directory
    docker run -v $PWD/certs:/certs -e SSL_SUBJECT="rancherdev.yourdomain.com" paulczar/omgwtfssl
    
  8. Optionally, you can now delete this container/image from Docker
  9. Run the following command to start Rancher in a Docker container (with persistent storage and custom SSL certificate)
    docker run -d -v /mnt/docker/mysql:/var/lib/mysql -v $PWD/rancher:/var/lib/rancher -v $PWD/certs/cert.pem:/etc/rancher/ssl/cert.pem -v $PWD/certs/key.pem:/etc/rancher/ssl/key.pem -v $PWD/certs/ca.pem:/etc/rancher/ssl/cacerts.pem --restart=unless-stopped -p 8080:80 -p 8443:443 rancher/rancher
    
  10. In order to internally resolve the custom rancherdev domain in RancherOS, add a loopback record it to the hosts file
    echo "127.0.0.1 rancherdev.yourdomain.com" | sudo tee -a /etc/hosts > /dev/null
    
  11. Rancher should now be running on the VM's public IP (run "ifconfig" to get your VM IP if you don't have it already)
  12. On your host OS (e.g. Windows) add this IP to the hosts file against "rancherdev.yourdomain.com" (c:\windows\system32\drivers\etc\hosts)
  13. Browse to the https://rancherdev.yourdomain.com:8443 in your web browser
  14. Follow the wizard to setup password/servername etc. for Rancher
Create a new Kubernetes cluster using Rancher
  1. In the Rancher browser UI - select to add a new cluster
  2. Choose "Custom" and use all the defaults, no cloud provider, [I disabled recurring etcd snapshots in the advanced options since this is a dev setup] - click Next
  3. In the next screen, choose all the Node Roles (etcd, Control Plane, Worker) - expand Advanced options and set the public and internal address to be 127.0.0.1 to ensure the node can survive an external IP change (or another copy running)
  4. Copy the generated Docker command to the clipboard and press Done - it should look something like this:
    sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.1.6 --server https://rancherdev.yourdomain.com:8443 --token XXX --ca-checksum XXX --node-name my-dev-node --address 127.0.0.1 --internal-address 127.0.0.1 --etcd --controlplane --worker
  5. Paste and run the command in the RancherOS shell
  6. Rancher should then provision the Kubernetes cluster
NB. Any links generated by the Rancher UI to containers you install will use "127.0.0.1" as the URL which is of course wrong from your host OS. You will need to manually enter the URL as rancherdev.yourdomain.com Surving an IP Change If you fire up the VM for the first time on another machine or your DHCP recycles and your external IP changes, you will need to follow these steps to get up and running:
  1. Run the VM as normal in Hyper-V
  2. Login via the Hyper-V console with rancher/yourpassword
  3. Get the IP address of the running RancherOS
    ifconfig
  4. Update your Windows host file (c:\windows\system32\drivers\etc\hosts) with and entry for rancherdev.yourdomain.com pointing to the VM IP
  5. Browse to the rancher URL and give it some time to come back online
Permalink: Setting up a Kubernetes cluster using Rancher on RancherOS

StackExchange.Redis Wrapper for JSON Chunking

January 07, 2019

I have used Redis caching with the StackExchange.Redis client in .NET across various projects and each time I find myself solving the same problems. The main problem, aside from abstracting the client and solving a few other issues (see below), is usually that my JSON data is bigger than Redis would like and it starts to perform badly or throws errors because the "qs" is full. I know there are other serialisation formats to try which might save some space, but my preference is to continue with JSON. I have created a GitHub repository called ChunkingRedisClient, which wraps up this boilerplate functionality in a central place. You can also install the current build as a NuGet package. Below is the write-up from the README: ---
# Chunking Redis Client
A library which wraps the StackExchange.Redis client, specifically using JSON serialisation, and adds functionality such as chunked reading/writing and sliding expiration.

The purpose of this library is to create a re-usable library of code (NB. which I need to put into a NuGet package) for wrapping the StackExchange.RedisClient and solving the issues I usually need to solve.

Those being:

* IoC wrappers/abstractions
   - Just take your dependency on "IRedisClient<TKey, TItem>"
   - By default you should configure your DI container to inject the provided RedisClient<TKey, TItem>
   - Since IoC is used throughout you also need to configure:
     ~ IRedisWriter<TKey, Item> -> JsonRedisWriter or ChunkedJsonRedisWriter
     ~ IRedisReader<TKey, Item> -> JsonRedisReader or ChunkedJsonRedisReader
     ~ IRedisWriter<TKey, Item> -> JsonRedisDeleter or ChunkedJsonRedisDeleter
     (note: for one combination of TKey, TItem - ensure the decision to chunk or not is consistent)
     ~ IKeygen<TKey> to an object specific implementation, like GuidKeygen
     ~ For chunking, locking is required:
             IRedisLockFactory -> RedisLockFactory
             To override the default of InMemoryRedisLock, call RedisLockFactory.Use<IRedisLock>() <-- your class here
     
* Strongly typed access to the cache
  - Use any C# object as your TKey and TItem, given that:
      ~ Your TKey is unique by GetHashCode(), or implement your own Keygen
      ~ Your TItem is serialisable by Newtonsoft.Json
      
* Implementing the StackExchange Connection Multiplexer
  - This is handled by the RedisDatabaseFactory
  - Not using the usual "Lazy<ConnectionMulitplexer>" approach, as I want to support one multiplexer per connection string (if your app is dealing with more than 1 cache)
  - The multiplexers are stored in a concurrent dictionary where the connection string is the key
  - The multiplexer begins connecting asynchronously on first use
    
* Sliding expiration of cache keys
  - Pass in the optional timespan to read methods if you want to use sliding expiration
  - This updates the expiry when you read the item, so that keys which are still in use for read purposes live longer
  
* Chunked JSON data
  - This solves a performance issue whereby Redis does not perform well with large payloads.
  - Sometimes you may also have had errors from the server when the queue is full.
  - The default chunk size is 10KB which can be configured in the ChunkedJsonRedisWriter
  - The JSON data is streamed from Newtonsoft into a buffer. Every time the buffer is full it is written to Redis under the main cache key with a suffix of "chunkIndex"
  - The main cache key is then written to contain the count of chunks, which is used by the reader and deleter.
  
* Generating keys for objects
  - I don't like using bytes for keys as they are not human readable, so I like to generate unique strings
  - There is no none-intrusive way of providing a type agnostic generic keygen, therefore you must write your own. If you write something for a CLR type, considering contributing it to the project!
  - Since we know Guids are unique, I have demonstrated the ability to create custom keygens.


The code can be extended to support other serialisation types (TODO), distributed locks (TODO), different ways of generating keys or whatever you need it to do.
Permalink: StackExchange.Redis Wrapper for JSON Chunking

Equality for Value Objects and Entities in DDD

December 12, 2018

In DDD most objects can be categorised as either value types or entities. Value types being objects where there is not one identifier, but simply a collection of related properties; entities being where the ID of the object is the ultimate identifier and all other properties are attributes of this entity. For me, the desired functionality in terms of equality comparisons is that entities are "Equal" when they have the same ID.. Value types are equal when they have matching "composite key" - i.e. all the properties of the object. To model this I have created a base class for enforcing value equality and a more specialised base for an entity:
public abstract class ValueEqualityObject<T> : IEquatable<T>
{
    public sealed override bool Equals(object obj)
    {
        if (obj is null)
            return false;

        if (ReferenceEquals(obj, this))
            return true;

        if (GetType() != obj.GetType())
            return false;

        return Equals((T)obj);
    }

    public sealed override int GetHashCode()
    {
        return TupleBasedHashCode();
    }

    public abstract bool Equals(T other);

    protected abstract int TupleBasedHashCode();
}

public abstract class Entity<TId> : ValueEqualityObject<Entity<TId>>
    {
        protected Entity(TId id)
        {
            Id = id;
        }

        public TId Id { get; }

        protected override int TupleBasedHashCode()
        {
            return (Id).GetHashCode();
        }

        public override bool Equals(Entity<TId> other)
        {
            return other != null 
                && other.Id.Equals(Id);
        }
    }
Now for each domain type I can choose which base to inherit from. For entities I simply define the ID, for value types I am prompted to define a TupleBasedHashCode and Equals method. The TupleBasedHashCode is a reminder to myself on a my preferred strategy for GetHashCode which is use the built-in Tuple implementation :)
Permalink: Equality for Value Objects and Entities in DDD

Lodash Memoize Wrapper for Caching Multiple Args

July 31, 2018

The Lodash memoize function caches a function call and can vary the cache items based on the parameters. By default the "cache key" is the first parameter, but often it's useful to vary by all parameters. Here is a simple wrapper that will use a custom resolver to always cache based on all args passed to the function. With this code in place, simply import this file instead of lodash version into your consuming code.
import _memoize from 'lodash-es/memoize';

export default function memoize(func)
{
    const resolver = (...args) => JSON.stringify(args);

    return _memoize(func, resolver);
}
Permalink: Lodash Memoize Wrapper for Caching Multiple Args

Batching Async Calls

April 25, 2018

To reduce the payload size of individual calls when loading up resources by ID, sometime you want to send multiple async requests in smaller batches. If we don't need to worry about local/remote resources (i.e. don't need intelligent partitioning or resource friendly approach), the easiest way is to fire off a load of tasks which consume a small batch from the superset. Here is a simple re-usable implementation:
public class BatchContentRequestor<TId, TValue>
{
    private readonly int _batchSize;
    private readonly Func<IEnumerable<TId>, Task<IEnumerable<TValue>>> _getContentAsyncFunc;

    public BatchContentRequestor(int batchSize, Func<IEnumerable<TId>, Task<IEnumerable<TValue>>> getContentAsyncFunc)
    {
        if (batchSize <= 0)
        {
            throw new ArgumentOutOfRangeException(nameof(batchSize), "Batch size must be a positive integer value.");
        }

        _batchSize = batchSize;
        _getContentAsyncFunc = getContentAsyncFunc ?? throw new ArgumentNullException(nameof(getContentAsyncFunc));
    }

    public async Task<IEnumerable<TValue>> GetContentBatchedAsync(IEnumerable<TId> allContentIds)
    {
        var allContentIdsList = allContentIds?.ToList();

        if (allContentIdsList == null || !allContentIdsList .Any())
        {
            return await _getContentAsyncFunc(allContentIdsList );
        }

        var allContentValues = new List<TValue>();

        var getBatchTasks = new List<Task<IEnumerable<TValue>>>();
        for (var batchStart = 0;
            batchStart < allContentIdsList.Count;
            batchStart += _batchSize)
        {
            var batchIds = allContentIdsList
                .Skip(batchStart)
                .Take(_batchSize)
                .ToList();

            getBatchTasks.Add(_getContentAsyncFunc(batchIds));
        }

        await Task.WhenAll(getBatchTasks).ConfigureAwait(false);

        foreach (var completedBatch in getBatchTasks)
        {
            allContentValues.AddRange(await completedBatch.ConfigureAwait(false));
        }

        return allContentValues;
    }
}
You can call it with your superset and it will automatically hit your callback function with the batches of IDs, will collect the results and return the superset of values. If the calling code passes a null or empty value this will still be passed to your callback for handling, making this a transparent proxy for the calling code. e.g.
var items = await new BatchContentRequestor<int, Item>(10, GetItemsByIdAsync).GetContentBatchedAsync(allItemIds).ConfigureAwait(false);
Permalink: Batching Async Calls