One idea I have been thinking about recently is the number of data transformations that happen within your network automation solution. The significance of them supports my long-running thesis that essentially all network automation is is moving data between systems and transforming it along the way.

We will start by looking at real-life examples of data transformations in network automation, and compare popular tools in the space and how they implement the patterns discovered in a second section.

A Real Life Example

A case I have observed multiple times in different environments is the following:

1. Schema A: System of record (SoR)
2. Schema B: Network source of truth (NSoT)
3. Schema C: Network controller
4. Schema D: Network device configuration

This means that a single piece of data would need to pass through three distinct transformations on its journey through this network automation solution.

1. Transforming the original schema from the SoRs into the schema of your NSoT
2. Passing intent from the NSoT to the device controller(s)
3. Transforming the controller’s internal configuration representation into the concrete configuration on the network device

Intuitively, this feels somewhat over-complicated. Let’s break it down.

If you should find any mistakes in my analysis, please let me know. You can find my contact details on the about page of this blog.

Transformation 1 – SoR to NSoT

The justification for this first transformation is essentially also the justification for having an NSoT in the first place. That is, in my opinion, to have trustable, aggregated data across as many facets of your network as possible inside a single data store with a reasonable API. Theoretically you might interface with all the APIs of your SoRs directly when aggregating your intent for the next transformation. Sticking with the NSoT justification, this would require your SoRs to have reliable data themselves. However, you would then lose out on almost all of the other goals of having a NSoT. I have, however, never seen anyone actually implement this approach while also applying reasonable software development practices like factoring out API clients into separate libraries.

Transformation 2 – NSoT to Controller

This is a transformation that, depending on your concrete network automation architecture, might not be happening in your environment. It is entirely possible to transform the intent in your NSoT directly into the network device configuration syntax. Network to Code (where I previously worked) has an excellent piece of software for this use case with the Nautobot Golden Config App. At the same time you might have many good reasons to inject a controller in the data path, such as

• Device can’t be managed without controllers (e.g. Cisco ACI)
• Separation of concerns between organizational units, possibly due to regulatory reasons
• Additional controller features that cannot (easily) be provided without them
• Cost of rebuilding your logic that takes the intent to the controller schema and finally to the device config without the intermediary step

If you have such a reason, you will have to deal with the consequences of having the additional data transformation step. Even if you were building the controller yourself, you probably don’t want to couple its data model with that of your NSoT, because otherwise they would end up as very tightly coupled components.

Transformation 3 – Controller to Device Config

This is the final transformation necessary to actually change a device’s configuration. It is also the one over which you (typically) have the least control, because it will likely happen in proprietary vendor software.

Honorable Mentions – Bonus Transformations

There are some further possible transformations that don’t lie in the direct path from SoR to device config. The purpose of this section is just to highlight additional instances of schema transformations, further driving home the point of how data and data transformation are very central concepts to a network automation solution.

Bilingual Controllers

Apparently Cisco recommends not using the NSO HTTP API for bulk changes, but rather to use the CLI. Since this is a schema different from that of the HTTP API, it means yet another transformation from the NSoT intent to a controller schema.

Monitoring Systems / Observability

Whether you are dealing with traditional network monitoring software, a modern observability type system, or something entirely different, your NSoT schema will need to be transformed into the schema used to configure targets in that software.

Managing Transformations

Now that we have established the importance of data transformations, let’s look at how different NSoT implementations are addressing the need for this. In general, I advocate for using standard solutions here in favor of integrating directly yourself with (HTTP) APIs.

Keeping Data in Sync

Paraphrasing Greenspuns’s tenth rule: Any sufficiently complicated data synchronization tool contains an ad hoc, informally-specified, bug-ridden, slow implementation of half diffsync or a similar tool. In all seriousness, there are a number of things you will inevitably re-implement yourself if you do, which are:

• Correctly diffing your data
• Providing useful error handling
• Create, update and delete logic for most, if not all, of your models

I will compare the three biggest relevant open-source tools in the space, specifically

• NetBox
• Nautobot
• Infrahub

Syncing Data from SoRs

• NetBox has Diode. I haven’t used this one, so I can’t comment on it.
• Nautobot has the SSoT app. I maintained this while working for NTC, so naturally I think it’s pretty cool software.
• Infrahub has Infrahub sync which, similar to the Nautobot SSoT App, is an NSoT-specific wrapper around diffsync.

Syncing Data to Systems

Both the SSoT app from Nautobot and Infrahub Sync can also be used in the other direction, that is, syncing data from the NSoT to a different system. There are a number of integrations available, but my personal opinion is that you are unlikely to benefit from those because they likely won’t match your specific schema requirements. Still, if you require ongoing synchronization between the two, I would highly recommend you choose one of these tools. For NetBox I wasn’t able to find anything for this specifically.

Making Transformations Explicit

Another interesting feature of an NSoT can be to explicitly handle data transformations (rather than handling them elsewhere, i.e. in your workflows, scripts, etc.). At its simplest, this means field-level transformations, implemented as Nautobot computed fields and Infrahub computed attributes (I wasn’t able to find anything for NetBox). One step up the hierarchy we can find Nautobot export templates, NetBox export templates, and Infrahub transformations. The first two are very similar (and probably mostly share the same codebase). The Infrahub implementation brings some novel ideas to the table. It integrates with the Infrahub concept of an artifact, which to me is the embodied idea of the result of a data transformation. In contrast to export templates, artifacts

• can be persisted in storage out of the box
• can reference other artifacts
• can be subject to approval through proposed changes, a concept which exists as NetBox Change Management and the (AFAIK now proprietary) Nautobot Version Control App, but is not extended to export template output

As you can probably tell at this point, I like the way it’s done in Infrahub. Disclaimer: I have not used this outside of a local lab context.

Persisting Transformation Output

There is a further kind of data transformation which I briefly want to address, and this is the idea of persisting the outcome of your transformations in your NSoT. We already talked about Infrahub artifacts, which do this already, but they just persist a JSON blob. Theoretically you could very easily implement this into Nautobot, NetBox, or your own custom solution by adding a table to store JSON data along with some metadata. This, however, is not what I consider persisting the outcome of your transformations, because it stores them at a very abstract level, rather than deeply integrating with your NSoT. This entails manipulating objects in your NSoT out of input data and/or existing objects, according to a certain predefined rule set. Furthermore, this concept includes lifecycle tracking through some sort of glue object that tracks which objects were created through the transformation and enables updating or deleting that output where necessary.

Christian Adell calls this “Design-Driven” in his work at designingnetworkautomation.com, which is in line with the Nautobot Design Builder App which implements this pattern. NetBox seems to have a proprietary offering in this space, which I wasn’t able to review as there is no public documentation beyond this blog post. Finally, Infrahub offers Generators. These also seem to check all the boxes for persisted transformation output, but my lab work with Infrahub hasn’t yet gone far enough to confirm this.

Summing Up

My personal opinion is that it is necessary to keep track of the data transformations happening in your network automation solution to be able to stick with a clean architecture through requirements that are bound to change eventually. This blog post should serve as an overview without any claim to comprehensiveness. Therefore, if you find anything is missing, please let me know!