Terraform `cidrsubnet` Deconstructed

When managing cloud Infrastructure as Code, network resources are an important element of your Terraform modules. Within your virtual network, you will need to carve out subnets depending on the access you want to permit to resources. Terraform provides a builtin function for easily defining subnets based on the virtual network CIDR block. Unfortunately the cidrsubnet() documentation is a little light, and the function can be a little intimidating to use. I’m going to break it down and give examples of how it can be used to make your subnet IP management easier.

Justin Ellingwood has a great post over on the Digital Ocean talking about IP Address, Subnets and CIDR Notation if you want the gritty details of subnet masking.

In a CIDR Network, (take 100.121.0.0/20 as an example) there are X number of subnets depending on the size of the subnet. Managing the subnet definitions allows for the most efficient use of your IP space in your network.

The cidrsubnet() function has the following signature:

cidrsubnet(iprange, newbits, netnum)

where

  • iprange is the CIDR block of your virtual network,
  • newbits is the new mask for the subnet within the virtual network, and
  • netnum is the zero-based index of the subnet when the network is masked with the newbit.

Here are a few examples to demonstrate the results from a cidrsubnet() function:

$ terraform console
> cidrsubnet("100.121.0.0/20", 8,64)
100.121.4.0/28
> cidrsubnet("100.121.0.0/20", 8,128)
100.121.8.0/28
> cidrsubnet("100.121.0.0/20", 5,4)
100.121.2.0/25
> cidrsubnet("100.121.0.0/20", 5,10)
100.121.5.0/25
> cidrsubnet("100.121.0.0/16", 12,2)
100.121.0.32/28

What is newbits?

The first argument I had to decipher was newbits, as it is a term I’ve never heard in relation to IP addressing. I understood it was related to the network mask for the subnet, but I didn’t immediately see the relationship to the binary netmasking I was familiar with. But after staring at results I recognized the pattern. The newbits was the difference between the virtual network netmask (/20 or /16 in the above examples) and the desired netmask (/28 or /25 in the above examples). Once I saw it, it became glaringly obvious and it is embarrassing to admit I didn’t immediately understand.

What is netnum?

When dividing a network into subnets, the number of subnets available is directly related to the size of the subnet and the size of the original network. By incrementing the netnum in the cidrsubnet() function we can declare our subnets on known boundaries programmatically.

The handy ipcalc tool is great for helping to visualize this, and determine how many subnets you have available. The tool works both via the website or as an installed binary. I find the local tool to be most useful.

With a couple quick ipcalc commands ($ ipcalc 100.121.0.0/16 /28 && $ ipcalc 100.121.0.0/20 /28) I can see that dividing a /16 into subnets of size /28 results in 4096 subnets, while a /20 will have 256 /28 subnets, since the /20 is a smaller network to begin with.

Here is a truncated example of the ipcalc output:

$ ipcalc 100.121.0.0/20 /25
Address:   100.121.0.0          01100100.01111001.0000 0000.00000000
Netmask:   255.255.240.0 = 20   11111111.11111111.1111 0000.00000000
Wildcard:  0.0.15.255           00000000.00000000.0000 1111.11111111
=>
Network:   100.121.0.0/20       01100100.01111001.0000 0000.00000000
HostMin:   100.121.0.1          01100100.01111001.0000 0000.00000001
HostMax:   100.121.15.254       01100100.01111001.0000 1111.11111110
Broadcast: 100.121.15.255       01100100.01111001.0000 1111.11111111
Hosts/Net: 4094                  Class A

Subnets after transition from /20 to /25

Netmask:   255.255.255.128 = 25 11111111.11111111.11111111.1 0000000
Wildcard:  0.0.0.127            00000000.00000000.00000000.0 1111111

 1.
Network:   100.121.0.0/25       01100100.01111001.00000000.0 0000000
HostMin:   100.121.0.1          01100100.01111001.00000000.0 0000001
HostMax:   100.121.0.126        01100100.01111001.00000000.0 1111110
Broadcast: 100.121.0.127        01100100.01111001.00000000.0 1111111
Hosts/Net: 126                   Class A

 2.
Network:   100.121.0.128/25     01100100.01111001.00000000.1 0000000
HostMin:   100.121.0.129        01100100.01111001.00000000.1 0000001
HostMax:   100.121.0.254        01100100.01111001.00000000.1 1111110
Broadcast: 100.121.0.255        01100100.01111001.00000000.1 1111111
Hosts/Net: 126                   Class A

 3.
Network:   100.121.1.0/25       01100100.01111001.00000001.0 0000000
HostMin:   100.121.1.1          01100100.01111001.00000001.0 0000001
HostMax:   100.121.1.126        01100100.01111001.00000001.0 1111110
Broadcast: 100.121.1.127        01100100.01111001.00000001.0 1111111
Hosts/Net: 126                   Class A

... <snip> ...

 31.
Network:   100.121.15.0/25      01100100.01111001.00001111.0 0000000
HostMin:   100.121.15.1         01100100.01111001.00001111.0 0000001
HostMax:   100.121.15.126       01100100.01111001.00001111.0 1111110
Broadcast: 100.121.15.127       01100100.01111001.00001111.0 1111111
Hosts/Net: 126                   Class A

 32.
Network:   100.121.15.128/25    01100100.01111001.00001111.1 0000000
HostMin:   100.121.15.129       01100100.01111001.00001111.1 0000001
HostMax:   100.121.15.254       01100100.01111001.00001111.1 1111110
Broadcast: 100.121.15.255       01100100.01111001.00001111.1 1111111
Hosts/Net: 126                   Class A


Subnets:   32
Hosts:     4032

So, the netnum would be the subnet number exposed there, minus 1 because of indexing differences. (Note in the output from the web-based CGI ipcalc the subnet numbers are not displayed.)

$ terraform console
> cidrsubnet("100.121.0.0/20", 5, 30)
100.121.15.0/25

IPAM with Terraform. Can it even?

Now it can certainly be argued that IP Address Management isn’t relevant in the Cloud. That it is not Cloudy to pre-define subnet sizes or blocks. And that is certainly true in many cases. Making an argument for why you might want to do it is a topic for another post.

Terraform makes it so easy to codify our networking, and provides easy reuse of submodules. We can abstract away some of the nitty gritty for the developer who just wants to quickly spin up a cloud instance or 1000. They don’t care about the subnet numbering.

A very simple use of the cidrsubnet() function would be to calculate a cidr_block value for a subnet resource when you want count number of subnets defined in consecutive IP space:

 cidr_block = "${cidrsubnet("100.121.0.0/20", 8, count.index )}"

assuming count = 3 would give us the subnets:

  • 100.121.0.0/28
  • 100.121.0.16/28
  • 100.121.0.32/28

Cloud Platform ACLs (aka Security Groups, Security Lists, Firewall Rules) are assigned to subnets within your virtual network. Because of this it makes the most sense to keep your subnets sized reasonably to keep the blast radius of access limited. Usually we would want very small subnets for things like load balancers, while the underlying servers would be in a larger subnet, as they would have different ACLs applied.

And this is where we start caring about the subnet addressing. When the IP space starts to overlap, defining the CIDR blocks can became a game of hunt-n-peck to find a usable block.

Example of overlapping subnets:

$ ipcalc 100.121.0.0/20 /25
 7.
Network:   100.121.3.0/25       01100100.01111001.00000011.0 0000000
HostMin:   100.121.3.1          01100100.01111001.00000011.0 0000001
HostMax:   100.121.3.126        01100100.01111001.00000011.0 1111110
Broadcast: 100.121.3.127        01100100.01111001.00000011.0 1111111
Hosts/Net: 126                   Class A

If the subnet above 100.121.3.0/25 is already carved out of your virtual network, the following /28 subnets would overlap, and not be available for use.

$ ipcalc 100.121.0.0/20 /28

49.
Network:   100.121.3.0/28       01100100.01111001.00000011.0000 0000
HostMin:   100.121.3.1          01100100.01111001.00000011.0000 0001
HostMax:   100.121.3.14         01100100.01111001.00000011.0000 1110
Broadcast: 100.121.3.15         01100100.01111001.00000011.0000 1111
Hosts/Net: 14                    Class A

 50.
Network:   100.121.3.16/28      01100100.01111001.00000011.0001 0000
HostMin:   100.121.3.17         01100100.01111001.00000011.0001 0001
HostMax:   100.121.3.30         01100100.01111001.00000011.0001 1110
Broadcast: 100.121.3.31         01100100.01111001.00000011.0001 1111
Hosts/Net: 14                    Class A

 51.
Network:   100.121.3.32/28      01100100.01111001.00000011.0010 0000
HostMin:   100.121.3.33         01100100.01111001.00000011.0010 0001
HostMax:   100.121.3.46         01100100.01111001.00000011.0010 1110
Broadcast: 100.121.3.47         01100100.01111001.00000011.0010 1111
Hosts/Net: 14                    Class A

 52.
Network:   100.121.3.48/28      01100100.01111001.00000011.0011 0000
HostMin:   100.121.3.49         01100100.01111001.00000011.0011 0001
HostMax:   100.121.3.62         01100100.01111001.00000011.0011 1110
Broadcast: 100.121.3.63         01100100.01111001.00000011.0011 1111
Hosts/Net: 14                    Class A

 53.
Network:   100.121.3.64/28      01100100.01111001.00000011.0100 0000
HostMin:   100.121.3.65         01100100.01111001.00000011.0100 0001
HostMax:   100.121.3.78         01100100.01111001.00000011.0100 1110
Broadcast: 100.121.3.79         01100100.01111001.00000011.0100 1111
Hosts/Net: 14                    Class A

 54.
Network:   100.121.3.80/28      01100100.01111001.00000011.0101 0000
HostMin:   100.121.3.81         01100100.01111001.00000011.0101 0001
HostMax:   100.121.3.94         01100100.01111001.00000011.0101 1110
Broadcast: 100.121.3.95         01100100.01111001.00000011.0101 1111
Hosts/Net: 14                    Class A

 55.
Network:   100.121.3.96/28      01100100.01111001.00000011.0110 0000
HostMin:   100.121.3.97         01100100.01111001.00000011.0110 0001
HostMax:   100.121.3.110        01100100.01111001.00000011.0110 1110
Broadcast: 100.121.3.111        01100100.01111001.00000011.0110 1111
Hosts/Net: 14                    Class A

 56.
Network:   100.121.3.112/28     01100100.01111001.00000011.0111 0000
HostMin:   100.121.3.113        01100100.01111001.00000011.0111 0001
HostMax:   100.121.3.126        01100100.01111001.00000011.0111 1110
Broadcast: 100.121.3.127        01100100.01111001.00000011.0111 1111
Hosts/Net: 14                    Class A

With a little upfront planning on subnet allocations, using Terraform to codify the boundaries, we can easily take the guess work out of declaring subnets, and ensure we have an efficient use of IP space.

A recent effort

I have recently implemented an approach for managing cloud platform subnets based on some policies and recommendations from our Networking team. We have a hybrid deployment, and for $reasons we limit the number of virtual networks in each of our cloud platform environments.

In one cloud provider platform we have virtual networks of size /17, which gives us 8 potential /20 subnets. This gives us some natural boundaries for allocating our actual subnets.

Within a /20 we have the following breakdowns available to mix and match

  • 512 /29 subnets; each with 6 Usable IPs (xsmall)
  • 256 /28 subnets; each with 14 Usable IPs (small)
  • 64 /26 subnets; each with 62 Usable IPs (medium)
  • 32 /25 subnets; each with 126 Usable IPs (large)

By allocating a /20 per subnet size we set some boundaries to prevent IP overlap, and start to define some variables that can be used in the cidrsubnet() calculation function.

variable subnet_allocation_map {
  description = "Map of CIDR blocks to carve into subnets based on size"
  type = map
  defautl = {
    xsmall = "100.121.0.0/20"
    small  = "100.121.144.0/20"
    medium = "100.121.160.0/20"
    large  = "100.121.176.0/20"
   }
}

variable "newbit_size" {
  description = "Map the friendly name to our subnet bit mask"
  type        = "map"

  default = {
    xsmall = "9"
    small  = "8"
    medium = "6"
    large  = "5"
  }
}

Based on what we know about the cidrsubnet(iprange, newbits, netnum) function, we are 2/3 of the way to a programmatic solution for defining subnets.

 cidr_block = "${cidrsubnet(
     lookup(var.subnet_allocation_map, "small"), 
     lookup(var.newbit_size,"small"),
     $netnum_to_be_calculated
)}"

Wherein my plan gets hairy

The rub here is the netnum value – if we want our subnets to be contiguous within our boundaries, that netnum needs to increment based on the already existing subnets.

I showed above how that is easily achieved via the count parameter to the subnet resource. But what about declaring additional subnets. Unfortunately my only answer to this is to count the existing subnets of $size and increment based on that. So the netnum argument becomes number_existing_subnets_of_this_size + count.index.

I’m going to pursue the the Terraform local values as a potential solution to easily find the existing subnet count. In the meantime I rely on Layer 8 intervention. Check back for an update on that.

If we already have 8 subnets of size small, and what to add a count of 3 more we can use the following parameter declaration:

cidr_block = "${cidrsubnet(
  lookup(var.subnet_allocation_map, "small"), 
  lookup(var.newbit_size,"small"),
   8 + count.index)}"

resulting in subnets of:

  • 100.121.0.128/28
  • 100.121.0.144/28
  • 100.121.0.160/28

Almost there

With the above cidr_block calculation using the cidr_block() function and a named set of subnets, we can build a re-useable module where developers simply pass in the size of the desired subnet as a parameter value. That is the goal anyway.

The built-in functions of Terraform can be very powerful, I encourage everyone to branch out from the index, lookup and map functions which are so familiar from programming languages. Explore the provided functions to see where you can abstract away some of the infrastructure management elements.