No Clean Feed - Stop Internet Censorship in Australia

Remotely-Triggered Black Hole (RTBH) Routing

Posted by stretch in None on Monday, 6 Jul 2009 at 5:05 a.m. GMT

Remotely-Triggered Black Hole (RTBH) routing is an interesting application of BGP as a security tool within service provider networks. One common use is mitigation of distributed denial of service (DDoS) attacks, as this article will explore.

Pictured below is a (very) simplified service provider architecture.

Topology

Routers 1 through 4 compose the network core, and router 9 functions as a standalone "management" router for route injection. OSPF is running across the core to exchange internal routes. Each router in this core square also maintains an iBGP adjacency with the other core routers, and with router 9. The server at 172.16.10.100 represents the target of a DDoS attack.

Assume a DDoS attack is launched from the public Internet toward the customer server at 172.16.10.100. The throughput consumed is so excessive that the attack is impacting the entire internal infrastructure and must be blocked at the edge. Due to the distributed nature of the attack, we must block at the edge all inbound traffic destined for the victim. Rather than resorting to laborious and error-prone access lists, we can utilize BGP and RTBH to quickly achieve the desired result.

Step 1: Null route preparation

The first two steps in configuring RTBH should ideally be completed prior to an attack.

RTBH works by injecting a specially-crafted BGP route into the network, forcing routers to drop all traffic with a specific next-hop -- effectively creating a "black hole." We create a static route on all BGP routers for this next-hop address:

R1(config)# ip route 192.0.2.1 255.255.255.255 Null0

This route forces any traffic destined for 192.0.2.1/32 to be immediately dropped by the router. This route is added to all edge routers (R1 and R2) in our example lab.

Note that any IP address can be used for this black hole route; we use an IP from the reserved Test-Net range (see RFC 3330) here out of convenience, as this IP should never appear on a routed network.

Step 2: Route-map preparation

As with the first step, this configuration should also be completed prior to an attack.

A route-map is created to redistribute certain tagged static routes into BGP with a modified next-hop value:

R9(config)# route-map RTBH
R9(config-route-map)# match tag 666
R9(config-route-map)# set ip next-hop 192.0.2.1
R9(config-route-map)# set origin igp
R9(config-route-map)# set community no-export

This is the key component to RTBH: any route advertised to an edge router with a next-hop of 192.0.2.1 will force recursion to the static Null0 route we implemented in the prior configuration, and any matching traffic will be dropped.

Enable static route redistribution into BGP for the route-map to take effect:

R9(config)# router bgp 65100
R9(config-router)# redistribute static route-map RTBH

Step 3: Create a victim route on the management router

Once an attack is detected and the decision is made to block traffic, a static route for the victim address is created on the management router (R9):

R9(config)# ip route 172.16.10.100 255.255.255.255 Null0 tag 666

Ideally, we would like to simply advertise this route to the edge BGP routers, but a route cannot be advertised as having an invalid next-hop. So, we've added a tag value to ensure that our RTBH route-map redistributes the route into BGP with a modified next-hop. Note that the no-export community has been appended here to avoid accidentally exporting the route beyond the local AS.

With our victim route injected, we can verify that the edge routers now drop all traffic bound for that prefix:

R1# show ip route 172.16.10.100
Routing entry for 172.16.10.100/32
  Known via "bgp 65100", distance 200, metric 0, type internal
  Last update from 192.0.2.1 00:06:14 ago
  Routing Descriptor Blocks:
  * 192.0.2.1, from 10.0.99.9, 00:06:14 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0

R1# show ip route 192.0.2.1
Routing entry for 192.0.2.1/32
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1

Of course, the victim is now unreachable, and we've effectively assisted the DDoS in accomplishing its goal. However we have protected our internal infrastructure (and other customers) from the flood of traffic, affording us time to better investigate and more eloquently mitigate the attack. As you might imagine, there are more advanced implementations of this method which can be used, as future articles will cover.

Thanks to Steinthor Bjarnason and his BRKSEC-2204 presentation at Cisco Live for inspiring this article!

Cisco Live schedule

Posted by stretch in Announcements on Friday, 26 Jun 2009 at 4:14 p.m. GMT

I was really hoping to have an article on control plane protection (CoPP) ready for today but I need to run some more extensive tests. In lieu of content I present my (tentative) Cisco Live schedule for next week:

Monday

  • 1:00 PM-3:00 PM - BRKAGG-1001 - Introduction to Cisco WiMAX Broadband System
  • 3:30 PM-5:30 PM - BRKRST-2264 - Multicast Maintenance
  • 5:30 PM-8:30 PM - World of Solutions 20th Anniversary Reception

Tuesday

  • 7:30 AM-9:30 AM - BRKARC-2001 - Cisco ASR1000 Series Routers' System & Solution Architectures
  • 10:00 AM-11:30 AM - GENKEY-5501 - Opening Keynote and Welcome Address with John Chambers
  • 12:00 PM-2:00 PM - BRKAGG-2004 - Architecting for IPv4-Exhaustion & IPv6 Deployment in Broadband Networks
  • 2:30 PM-3:30 PM - GENSSN-5641 - Partner Super Session: Intel
  • 4:00 PM-6:00 PM - BRKOPT-1200 - IPv6 security for Service Providers

Wednesday

  • 8:00 AM-10:00 AM - BRKSEC-2000 - Secure Enterprise Design
  • 10:30 AM-11:30 AM - GENKEY-5564 - Cisco Technology Keynote with Padmasree Warrior
  • 12:00 PM-2:00 PM - BRKSEC-2201 - Emerging Threats
  • 4:00 PM-6:00 PM - BRKSEC-2204 - What Can Enterprises Learn as Security Practice from Service Providers?
  • 7:30 PM-11:30 PM - Customer Appreciation Event

Thursday

  • 8:00 AM-10:00 AM - BRKRST-2362 - Deploying Performance Routing
  • 10:30 AM-11:30 AM - GENKEY-5565 - Closing Guest Keynote: Guy Kawasaki
  • 12:00 PM-2:00 PM - BRKARC-2002 - Network Diagnosis: Prevent, Prepare, Repair

Miscellaneous news

Posted by stretch in Announcements on Monday, 22 Jun 2009 at 10:27 a.m. GMT

A few quick announcements today...

First, the Packet Life RSS feed is now available on the Amazon Kindle for $0.99/mo. (This price is set by Amazon. If it were up to me it'd be free.)

Second, I've established an informal IRC channel for the site on the Freenode IRC network. Connect to irc.freenode.net and join #packetlife if interested. This is an experiment, to determine whether it generates useful discussion or simply a barrage of requests for braindumps. We'll see.

Finally, for those who haven't heard, I've been busy working on a new revision of the site. This next incantation aims to shift focus from my own work to forming a collaborative community. So far, I have in mind a community blog, wiki, forum, chat, and some resources already available on the site. I am open to suggestions and ideas for other things to throw in. I hope to have a functional beta of the site out in a couple months.

QoS pre-classification

Posted by stretch in Networking on Wednesday, 17 Jun 2009 at 10:37 a.m. GMT

Implementing quality of service provisions on virtual tunnel interfaces (VTIs) poses a challenge: QoS policy enforcement is normally applied only after tunnel encapsulation has taken place. Consider the following service policy applied to a GRE tunnel:

class-map match-all ICMP
 match access-group name MATCH_ICMP
class-map match-all GRE
 match access-group name MATCH_GRE
!
!
policy-map MyPolicy
 class ICMP
 class GRE
!
interface Tunnel0
 ip address 192.168.0.1 255.255.255.252
 tunnel source 10.0.0.1
 tunnel destination 10.0.0.2
!
interface FastEthernet0/0
 ip address 10.0.0.1 255.255.255.252
 service-policy output MyPolicy
!
ip access-list extended MATCH_GRE
 permit gre any any
ip access-list extended MATCH_ICMP
 permit icmp any any

(Note that the policy map used as an example here does not actually enforce any policy; its class maps are included simply to illustrate the manner in which packets are being matched.)

In default operation, our QoS policy will be applied to packets exiting the physical interface only after they have undergone tunnel encapsulation:

Default classification

When we generate ICMP traffic to the far end of the tunnel, we can verify that our policy only matches the packets as GRE traffic:

R1# ping 192.168.0.2 repeat 10

Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 8/10/24 ms
R1# show policy-map interface f0/0
 FastEthernet0/0 

  Service-policy output: MyPolicy

    Class-map: ICMP (match-all)
      0 packets, 0 bytes
      30 second offered rate 0 bps
      Match: protocol icmp

    Class-map: GRE (match-all)
      10 packets, 1240 bytes
      30 second offered rate 0 bps
      Match: protocol gre

    Class-map: class-default (match-any)
      5 packets, 620 bytes
      30 second offered rate 0 bps, drop rate 0 bps
      Match: any

This hinders our ability to differentiate between packets which have undergone tunnel encapsulation. However, enabling QoS pre-classification configures the tunnel interface to maintain a copy of the original packet in memory long enough for the physical interface policy to inspect it.

Pre-classification

After enabling pre-classification, our policy is applied to the original "inner" header instead of the encapsulation header:

R1# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)# interface tunnel0
R1(config-if)# qos pre-classify
R1(config-if)# ^Z
R1# clear counters f0/0
Clear "show interface" counters on this interface [confirm]
R1# ping 192.168.0.2 repeat 10

Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 4/8/12 ms
R1# show policy-map interface f0/0
 FastEthernet0/0 

  Service-policy output: MyPolicy

    Class-map: ICMP (match-all)
      10 packets, 1240 bytes
      30 second offered rate 0 bps
      Match: access-group name MATCH_ICMP

    Class-map: GRE (match-all)
      0 packets, 0 bytes
      30 second offered rate 0 bps
      Match: access-group name MATCH_GRE

    Class-map: class-default (match-any)
      2 packets, 120 bytes
      30 second offered rate 0 bps, drop rate 0 bps
      Match: any

A couple side notes from my experiments on IOS 12.4(22)T:

  • I initially wrote the class maps to utilize NBAR (match protocol), but for some reason the ICMP class map didn't appear to take effect, so I switched to extended ACLs. Your mileage may vary.

  • Pre-classification seems to have no effect when applied to a VTI performing native IPsec encapsulation (versus a crypto map). Again, this may be IOS version-specific.

Addressing the elephant in the tubes

Posted by stretch in Opinion on Thursday, 11 Jun 2009 at 3:43 p.m. GMT

Cisco recently released a report predicting massive growth in Internet video traffic over the next several years. Arguably the most provocative prediction made therein states that "global IP traffic will quintuple from 2008 to 2013."

The Internet, to borrow a line from Douglas Adams, is big. Really big. So an estimated 400% growth in its traffic in just five years is more than mild conjecture. The report goes on to assert that "Internet video is now approximately one-third of all consumer Internet traffic, not including the amount of video exchanged through P2P file sharing," which seems unlikely. Then again, when you consider the sheer infeasibility of accurately arriving at these numbers in the first place, any credibility of the report instantly deteriorates into that of a marketing brochure anyway.

Which isn't to say huge growth shouldn't be expected and welcomed; indeed, most experts agree that this Internet thing is here to stay. However, reports like this one, for all their abstract arithmetic, fail to address the ominous creep of artificial throughput limitations quietly (and not so quietly) being implemented by Internet service providers around the world.

Bandwidth caps have long been a burden of life in countries such as Australia and the UK, one which is slowly spreading to broadband subscribers across North America. Last year, Comcast announced an official 250 GB transfer limit amid much controversy regarding their definition of "acceptable use." Several Canadian ISPs have recently had their bandwidth throttling practices called into question. And of course there's the utter nonsense undertaken by Time Warner.

With such miserly tendencies becoming the norm, Internet growth is sure to fall far short of Cisco's predictions. The question now is whether pressure from network vendors and content providers will be enough to reverse the course undertaken by so many ISPs. There is no denying that the technology to support such increases in throughput is available, but can service providers be convinced to adopt it?

UPDATE: BT leadership apparently doesn't understand how the Internet works. But we're to believe that the business strategies at work here are top notch and residential transfer caps are simply a fact of life? Right.

The vendor-client relationship

Posted by stretch in Humor on Thursday, 4 Jun 2009 at 8:56 a.m. GMT

We engineers often revel in amusement at the expense of vendors. Here's one for those on the other side of the fence.

Thanks to @tonhe for pointing this out. Unfortunately there doesn't seem to be much information regarding the origin of the video; if anyone can shed some light on who made it I'd love to know.

Access list syslog correlation

Posted by stretch in Networking on Monday, 1 Jun 2009 at 6:25 a.m. GMT

ACL syslog correlation is a Cisco IOS feature which provides the ability to identify which access list entry (ACE) was responsible for a permit or deny action appearing in syslog.

Consider the following access list applied to filter externally-sourced traffic destined for the internal network:

Router# show ip access-lists
Extended IP access list EXTERNAL->INTERNAL
    10 permit icmp any any echo-reply
    20 permit udp any eq domain any
    30 permit tcp any any established
    40 permit tcp any host 172.16.0.202 eq www
    50 deny tcp any any eq 22 log
    999 deny ip any any

Lines 10 through 40 allow miscellaneous traffic types in, and line 50 explicitly denies inbound traffic destined to TCP/22 for the purpose of logging. Line 60 defines an explicit "deny any"; this isn't required, but included here for clarity.

When an inbound packet is matched and discard by rule 50, the following syslog message is generated:

%SEC-6-IPACCESSLOGP: list EXTERNAL->INTERNAL denied tcp 10.0.0.2(57289) ->
 10.0.0.1(22), 1 packet

This log message informs the administrator of the action taken and the ACL which matched the packet, but in some cases we may want more detail readily available. This is where ACL hash correlation comes in handy. We can redefine the ACL rule to include an arbitrary keyword, up to 64 characters in length:

Router(config)# ip access-list ext EXTERNAL->INTERNAL
Router(config-ext-nacl)# 50 deny tcp any any eq 22 log Deny_SSH
Router(config-ext-nacl)# ^Z
Router# show ip access-lists
Extended IP access list EXTERNAL->INTERNAL
    10 permit icmp any any echo-reply
    20 permit udp any eq domain any
    30 permit tcp any any established
    40 permit tcp any host 172.16.0.202 eq www
    50 deny tcp any any eq 22 log (tag = Deny_SSH)
    999 deny ip any any

Now when that rule discards a packet, its log message includes our tag:

%SEC-6-IPACCESSLOGP: list EXTERNAL->INTERNAL denied tcp 10.0.0.2(54843) ->
 10.0.0.1(22), 1 packet  [Deny_SSH]

These tags can be reused by multiple ACEs. Continuing our example, say we also want to deny traffic to SSH servers running on an alternate port, TCP/2222:

Router(config)# ip access-list ext EXTERNAL->INTERNAL
Router(config-ext-nacl)# 60 deny tcp any any eq 2222 log Deny_SSH
Router(config-ext-nacl)# ^Z
Router# show ip access-lists
Extended IP access list EXTERNAL->INTERNAL
    10 permit icmp any any echo-reply
    20 permit udp any eq domain any
    30 permit tcp any any established
    40 permit tcp any host 172.16.0.202 eq www
    50 deny tcp any any eq 22 log (1 match) (tag = Deny_SSH)
    60 deny tcp any any eq 2222 log (tag = Deny_SSH)
    999 deny ip any any

ACL logging tags are great for readily identifying ACE matches, particularly when you expect certain matches to occur, but manually defining a tag for each ACE isn't always practical. This is especially true when you want to maintain syslog correlation among hundreds or thousands of individual ACL rules. In such cases, automatically generated hashes would be the preferred option. Automatic hashing is enabled globally, and applied to any ACE with the log keyword with no argument following it.

Here we enable hash generation and add a few more ACEs to our ACL:

Router(config)# ip access-list logging hash-generation
Router(config)# ip access-list extended EXTERNAL->INTERNAL
Router(config-ext-nacl)# 70 deny tcp any any eq 1000 log
Router(config-ext-nacl)# 80 deny tcp any any eq 2000 log
Router(config-ext-nacl)# 90 deny tcp any any eq 3000 log
Router(config-ext-nacl)# ^Z
Router# show ip access-lists
Extended IP access list EXTERNAL->INTERNAL
    10 permit icmp any any echo-reply
    20 permit udp any eq domain any
    30 permit tcp any any established
    40 permit tcp any host 172.16.0.202 eq www
    50 deny tcp any any eq 22 log (1 match) (tag = Deny_SSH)
    60 deny tcp any any eq 2222 log (tag = Deny_SSH)
    70 deny tcp any any eq 1000 log (hash = 0x594F8697)
    80 deny tcp any any eq 2000 log (hash = 0xE471528C)
    90 deny tcp any any eq 3000 log (hash = 0x34CC0DD6)
    999 deny ip any any

As expected, a trigger by any of the ACEs with a hash appended works just as it does with tags:

%SEC-6-IPACCESSLOGP: list EXTERNAL->INTERNAL denied tcp 10.0.0.2(21196) ->
 10.0.0.1(1000), 1 packet  [0x594F8697]

While not immediately useful, a hash can be used to filter an ACL configuration and quickly identify the ACE responsible for an action:

Router# show ip access-lists EXTERNAL->INTERNAL | include 594F8697
    70 deny tcp any any eq 1000 log (1 match) (hash = 0x594F8697)

IP source guard without DHCP

Posted by stretch in Networking on Monday, 25 May 2009 at 2:21 p.m. GMT

Discussion of IOS' IP source guard feature typically accompanies the configuration of DHCP snooping. This is to be expected, as IP source guard relies on a switch's knowledge of DHCP-assigned host addresses in order to validate and restrict spoofed source addresses. However, IP source guard can be implemented independent of DHCP, a useful ability on networks or subnets using only static addressing.

When DHCP snooping is enabled, a switch maintains a database of the DHCP addresses assigned to the hosts connected to each access port. IP source guard references this database when a packet is received on any of these interfaces and compares the source address to the assigned address listed in the database. If the source address differs from the "allowed" address, the packet is assumed to spoofed and is discarded.

Assuming DHCP isn't available or in use on a subnet, static IP bindings can be manually configured per access port to achieve the same effect. The following topology illustrates the lab on which this is being demonstrated.

Lab topology

The first step is to enable IP source guard on every access interface:

Switch(config)# interface f0/10
Switch(config-if)# ip verify source
Switch(config-if)# interface f0/20
Switch(config-if)# ip verify source

(Note that for the purposes of the lab, IP source guard has only been enabled on the two relevant access ports. In a real-world deployment, IP source guard should be enabled consistently across all access ports.)

The next step isn't immediately obvious, and in fact a bit counter-intuitive: enabling DHCP snooping. Despite our reliance solely on static bindings in this lab, the DHCP snooping feature must be turned on to enable the inspection of incoming packets. DHCP snooping must be enabled globally, and again for each VLAN on which IP source guard will be run.

Switch(config)# ip dhcp snooping
Switch(config)# ip dhcp snooping vlan 10

Next we'll define the static IP source address bindings, under global configuration. Note that this also requires the source MAC address, which can be obtained from the switch's CAM table if not already known.

Switch# show mac address-table int f0/10
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    001d.60b3.0add    DYNAMIC     Fa0/10
Total Mac Addresses for this criterion: 1
Switch# show mac address-table int f0/20
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    0023.7d00.d0a8    DYNAMIC     Fa0/20
Total Mac Addresses for this criterion: 1
Switch# conf t
Switch(config)# ip source binding 001d.60b3.0add vlan 10 10.0.0.10 interface f0/10
Switch(config)# ip source binding 0023.7d00.d0a8 vlan 10 10.0.0.20 interface f0/20

We can verify our new entries with show ip source binding:

Switch# show ip source binding
MacAddress          IpAddress     Lease(sec)  Type        VLAN  Interface
------------------  ------------  ----------  ----------  ----  -----------------
00:1D:60:B3:0A:DD   10.0.0.10     infinite    static       10    FastEthernet0/10
00:23:7D:00:D0:A8   10.0.0.20     infinite    static       10    FastEthernet0/20
Total number of bindings: 2

The above output displays the bindings active in the database. However, to inspect the actual operation of IP source guard, the command show ip verify source is used:

Switch(config)# do sh ip verify source
Interface  Filter-type  Filter-mode  IP-address       Mac-address     Vlan
---------  -----------  -----------  ---------------  --------------  ----
Fa0/10     ip           active       10.0.0.10                        10
Fa0/20     ip           active       10.0.0.20                        10

If Filter-mode above lists inactive-no-snooping-vlan for any entry, DHCP snooping has not been enabled for that VLAN (which I totally already knew and wasn't at all a mistake I made in developing this lab).

On host B, we can verify that a legitimate ping (sourced from 10.0.0.20) to host A succeeds:

Host_B$ ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_seq=1 ttl=64 time=0.465 ms
64 bytes from 10.0.0.10: icmp_seq=2 ttl=64 time=0.297 ms
64 bytes from 10.0.0.10: icmp_seq=3 ttl=64 time=0.385 ms

To verify that IP source guard is blocking spoofed traffic, we'll need to craft a custom packet. Scapy is a great tool for this:

Host_B# scapy
Welcome to Scapy (2.0.0.11 beta)
>>> sendp(Ether()/IP(src='1.2.3.4', dst='10.0.0.10')/ICMP(), iface='eth0')
.
Sent 1 packets.

The above command in the scapy interpreter sends an ICMP echo request to 10.0.0.10 sourced from 1.2.3.4. Although we're obviously unable to test for a response to this ping (even if it got through, the reply would be to 1.2.3.4 instead of Host B), running a protocol analyzer on host A verifies that the spoofed packet is never forwarded by the switch.

What's your preferred study method?

Posted by stretch in Announcements on Thursday, 14 May 2009 at 6:05 a.m. GMT

What's your preferred study method?

  • Text books (37%)
    Percentage of vote
  • Self-paced videos (30%)
    Percentage of vote
  • Virtual classroom (5%)
    Percentage of vote
  • Physical classroom (26%)
    Percentage of vote

I've been considering producing my own commercial training materials, and would like to get a better idea of what medium most people prefer to use. In the poll here I've listed the four methods I think are most common.

Text books - Includes any study guides or reference material in dead-tree format.

Self-paced videos - Prerecorded videos of training presentations (assume this covers audio-only recordings as well).

Virtual classroom - Live classes held virtually.

Physical classroom - Live classes held at a physical location.

If you have another you'd like to add please mention it in the comments below. Note that I'm only looking for actual training materials here; it is assumed that, like any networker worth his paycheck, you're labbing out whatever you're studying as well.

ARIN formally notifies CEOs of upcoming IPv4 depletion

Posted by stretch in Opinion on Monday, 4 May 2009 at 2:19 a.m. GMT

ARIN recently sent a letter to the CEOs of organizations which hold IPv4 addresses in North America to formally notify them of the approaching IPv4 address pool depletion. The letter, available here (PDF), begins:

IP addresses are the numbers behind domain names and are essential to the Internet. In May 2007, the American Registry for Internet Numbers (ARIN) advised the Internet community on IP address depletion in what is called Internet Protocol version 4 (IPv4). At the current rate of consumption, IPv4 will be depleted within the next two years. After that, organizations that need additional IP addresses will need to adopt IPv6, a newer version of the Internet Protocol that provides a much larger pool of address space.

It then goes on to explain that IPv6 should be adopted as soon as possible, and that from this point forward IPv4 space applications will face great scrutiny.

Personally, I don't care for the way this letter is worded. It reads like a letter from the cable company notifying customers of increased rates. Put yourself in the position of a typical CEO or vice president reading this letter, someone unfamiliar with Internet infrastructure, and it's natural to think, "Damn those tech wienies! Because they failed to plan ahead, I have to pay to upgrade my entire network!" Obviously there's a lot more substance to the issue, and there are considerable benefits to be reaped beyond acquiring a greater address space.

Following the quote above, the letter continues, "You should begin planning for IPv6 adoption if you are not doing so already." Seriously? We're well past the point of "you should begin thinking about this" and fast approaching "hope you're ready by now." IPv6 isn't new, and it isn't unexpected. Correspondence from registrars needs to reflect this if there's any hope of instilling a sense of urgency.

Of course, organizations which haven't yet at least formed an IPv6 strategy will be the ones kicking and screaming about the absence of new address space in a few years. But they still won't adopt IPv6. They'll NAT the hell out of their existing infrastructure and resist upgrading to new applications they can't support. In little time they will have spent more mitigating IPv6 than they would have had they adopted it responsibly.

Organizations need to embrace the fact that such an overhaul is a necessity of growth, not a symptom of poor planning. Upgrading IP is not some fabricated service tax, but a recurring operational cost. It's simply not obvious because, with a cycle of roughly forty years, this is the first time it has recurred.