Variadic Generic Types in Swift

I was recently helping a friend convert his Shopify order CSVs into a custom format used by his accounting software. If I didn’t like this friend as much, I probably wouldn’t have volunteered to do this. It is quite annoying work, as it involves various steps of parsing, validation, refining and transforming data. It is quite easy to get confused when you’re dealing with many data processing steps and at some point code will lose its semantic transparency, i.e., it is not immediately obvious what it does. Most developers will recognise this experience and will hopefully, barring immediate deadlines, refactor their code the moment they realise that this is happening. To me, the solution was to design a little pipeline framework. This way, I could declaratively define the different steps and have a better representation of the actual intention of the code.

As the title indicates, this article is not about parsing and transforming Shopify order CSVs. It is about Swift’s support for variadic generic types. The reason for this preamble is because I hope that data pipelines might provide an easy and intuitive example to learn about an abstruse topic. Everyone knows generics, but what does it mean to have variadic generics? Is this actually a valuable feature and what are real-world use-cases? That is what this article aims to explain.

The Problem at Hand

When I started writing out the data pipeline, I originally imagined it as an “object oriented” way to declaratively define complex function compositions. A Piece of the pipeline would be an object defined by some apply function which takes an Input and returns an Output:

protocol Piece<Input, Output> {
    associatedtype Input
    associatedtype Output
    
    func apply(_ input: Input) -> Output
}

In my mind a pipeline would then be a straight line from start to finish, slowly modifying and refining the source data. The pieces of the pipeline would correspond to the following steps: Parse, Clean, Refine and Transform:

I soon realised, however, that this was not adequate. In particular, transforming the data often meant transforming different parts of the data in domain specific and semantically disparate ways. (Perhaps we need to map our Shopify product ID to an internal product ID, or add some additional tax data based on order info.) As such these should not be represented by one large monolithic “transform” piece, but a modular way to define multiple transformations on the same object. I began reimagining the pipeline as such:

By allowing us to split the pipeline into multiple sub-pipelines, the definition of the pipeline bears closer resemblance to the conceptual model of what we’re actually doing. If we were doing this work manually, we would probably intuitively imagine each of these sub-pipelines as separate tasks that we need to complete in order to create our new accounting entries based on our Shopify CSV data.

This kind of pipeline is often referred to as a Directed Acyclic Graph pipeline or DAG pipeline for short. (Very prevalent in literature is simply referring to it as “a DAG”. I personally do not like this as it makes it appear as if all Directed Acyclic Graphs are pipelines, which is obviously untrue.)

The problem now is: How do we implement this Split piece? A cursory look at the diagram makes it clear that we cannot use normal generics, as a generic would require us to exhaustively define all possible amounts of output types. For a split with 1 sub-pipeline, i.e., Split1, it would be:

struct Split1<Input, P: Piece>: Piece
where P.Input == Input {
    let piece: P

    func apply(_ input: Input) -> P.Output {
        piece.apply(input)
    }
}

While for a Split2, it would be:

struct Split2<P1: Piece, P2: Piece>: Piece
where P1.Input == P2.Input {
    let piece1: P1
    let piece2: P2

    func apply(
	    _ input: P1.Input
    ) -> (P1.Output, P2.Output) {
        (piece1.apply(input), piece2.apply(input))
    }
}

Obviously we cannot exhaustively define an infinite amount of variants for this struct. In older libraries (even in SwiftUI), this problem would often be solved by using code generation to define all overloads of a function or variants of a struct up to a certain number of type parameters. (This is generally called that type’s -arity.) With the advent of variadic generics, barring some complex type constraints or behavioural requirements, this is no longer necessary.

The Basics of Variadic Generics

I will assume the reader’s familiarity with the concept of generic types. A generic type can be interpreted as a “type template”, it cannot be instantiated as is, but if the right type argument is provided, it can be specialised into a concrete type. A simple example of this is an Optional type:

enum Optional<Element> {
    case some(_ element: Element),
         none
}

Generics are useful because they allow greater code abstraction and reduce boilerplate code or code duplication. A variadic generic, then, can be interpreted as a further generalisation of a generic. Instead of having a fixed amount of type arguments that define the generic template, we can provide an arbitrary amount of type arguments.

An example of a variadic generic function would be something like this:

func render<each V: View>(views: (repeat each V)) {
    for view in repeat each views {
        view.render();
    }
}

Our Split Struct

It should now already be quite obvious how variadic generics help us tackle this issue in a clean way. Instead of defining a bunch of overloads, each having similar logic, we can define a variadic generic struct that is somewhat similar to this:

struct Split<Input, each P: Piece>: Piece
where repeat (each P).Input == Input {
    let pieces: (repeat each P)
    
    func apply(
	    _ input: Input
    ) -> (repeat (each P).Output) {
        (repeat (each pieces).apply(input))
    }
}

This however has one major issue: (each P).Input == Input is a “same element constraint” on the variadic generic P, and as of the time of writing (Swift 6), these are not yet supported by the Swift compiler or runtime. Does this mean we should shrug and say “well guess that’s it then”? This question is rhetorical because I wouldn’t have written an article if that was the case.

There is a very simple way around this problem. Instead of a simple Split struct, we instead use an accumulator builder pattern—the same thing often used when implementing result builders and, in fact, for my implementation I did decide to use a result builder for this. Instead of one big Split piece, internally it will look something like this:

Each Split accumulator only needs to ensure a same type contraint on two Pieces, as such it can be defined using a simple generic. The variadic part here is the Output of the split. Because the Output does not impose any same-type constraints, it can be easily defined using normal variadic type definitions. The SplitAccumulator is thus defined as:

struct SplitAccumulator<
    P1: Piece,
    P2: Piece,
    each P1Outputs
>: Piece
where
    P1.Output == (repeat each P1Outputs),
    P1.Input == P2.Input
{
    let piece1: P1
    let piece2: P2
    
    public func apply(
        _ input: P1.Input
    ) -> (repeat each P1Outputs, P2.Output) {
        let resultsAccumulated = piece1.apply(input)
        let resultNext = piece2.apply(input)
        
        return (
            repeat each resultsAccumulated,
            resultNext
        )
    }
}

The base case for this accumulator is the first piece passed to the Split accumulator.

The actual implementation of this struct will be a little different in practice, this is because runtime support for variadic generic structs requires a recent Swift runtime and requires, e.g., macOS 14.0 to work. To circumvent this restriction, we can instead inject variadic functions into the struct which do not have the runtime requirements. If you are curious about how this can be done, you can look at the repository here, which implements it with backward compatibility for older Swift runtimes.

With this piece implemented, the most complex part of our pipeline architecture should be sorted. All the other parts are now quite straightforward. To turn the various split outputs into a new data structure, we can simply define a Piece that takes the n-tuple Output as its Input and returns a new data type as its Output.

Conclusion

This article might seem be a little complicated still, but I hope that this complexity is due to the inherent complexity of variadic generics and not due to a lacking explanation. If we look back at our careers, we can all recognise how foreign generics felt when learning about them for the first time. Now, however, you’d be hard pressed to write any code without generics due to the obvious utility that they provide. A similar utility can be found when talking about variadic generics. They might seem foreign at first, but when they can be used, they often provide great benefit to the resulting architecture.