idnits 2.17.00 (12 Aug 2021) /tmp/idnits64513/draft-livingood-woundy-congestion-mgmt-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 764: '... congestion MAY occur soon....' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 10, 2010) is 4483 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force C. Bastian 3 Internet-Draft T. Klieber 4 Intended status: Informational J. Livingood, Ed. 5 Expires: August 14, 2010 J. Mills 6 R. Woundy 7 Comcast 8 February 10, 2010 10 Comcast's Protocol-Agnostic Congestion Management System 11 draft-livingood-woundy-congestion-mgmt-03 13 Abstract 15 This document describes the congestion management system of Comcast 16 Cable, a large cable broadband Internet Service Provider (ISP) in the 17 U.S. Comcast completed deployment of this congestion management 18 system on December 31, 2008. 20 Status of this Memo 22 This Internet-Draft is submitted to IETF in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF), its areas, and its working groups. Note that 27 other groups may also distribute working documents as Internet- 28 Drafts. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 The list of current Internet-Drafts can be accessed at 36 http://www.ietf.org/ietf/1id-abstracts.txt. 38 The list of Internet-Draft Shadow Directories can be accessed at 39 http://www.ietf.org/shadow.html. 41 This Internet-Draft will expire on August 14, 2010. 43 Copyright Notice 45 Copyright (c) 2010 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. Key Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 62 3. Historical Overview . . . . . . . . . . . . . . . . . . . . . 7 63 4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 64 5. Implementation and Configuration . . . . . . . . . . . . . . . 9 65 5.1. Thresholds For Determining When a CMTS Port Is in a 66 Near Congestion State . . . . . . . . . . . . . . . . . . 13 67 5.2. Thresholds For Determining When a User Is in an 68 Extended High Consumption State and for Release from 69 that Classification . . . . . . . . . . . . . . . . . . . 14 70 5.3. Effect of BE Quality of Service on Users&apos 71 Broadband Experience . . . . . . . . . . . . . . . . . . . 17 72 5.4. Equipment/Software Used and Location . . . . . . . . . . . 19 73 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 21 74 7. Exceptional Network Utilization Considerations . . . . . . . . 21 75 8. Security Considerations . . . . . . . . . . . . . . . . . . . 22 76 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 77 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 78 11. Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . 22 79 12. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 23 80 13. Informative References . . . . . . . . . . . . . . . . . . . . 23 81 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26 83 1. Introduction 85 Comcast Cable is a large broadband Internet Service Provider (ISP), 86 based in the U.S., serving the majority of its customers via cable 87 modem technology. During the late part of 2008, and completing on 88 December 31, 2008, Comcast deployed a new congestion management 89 system across its entire network. This new system was developed in 90 response to dissatisfaction in the Internet community as well as 91 complaints to the U.S. Federal Communications Commission (FCC) 92 regarding Comcast's old system, which targeted specific peer-to-peer 93 (P2P) applications. This new congestion management system is 94 protocol-agnostic, meaning that it does not examine or impact 95 specific user applications or network protocols, which is perceived 96 as a more fair system for managing network resources at limited times 97 when congestion may occur. 99 The purpose of this document is to describe how this example of a 100 large scale congestion management system functions. This is 101 primarily in response to questions from other ISPs as well as 102 solution developers, who are interested in learning from and/or 103 deploying similar systems in other networks. 105 2. Key Terminology 107 This section defines the key terms used in this document. Some terms 108 below refer to elements of the Comcast network. As a result, it may 109 be helpful to refer to Figure 1 when reviewing some of these terms. 111 2.1. Cable Modem 113 A device located at the customer premise used to access the Comcast 114 High Speed Internet (HSI) network. In some cases, the cable modem is 115 owned by the customer, and in other cases it is owned by the cable 116 operator. This device has an interface (i.e., someplace to plug in a 117 cable) for connecting the coaxial cable provided by the cable company 118 to the modem, as well as one or more interfaces for connecting the 119 modem to a customer's PC or home gateway device (e.g., home gateway, 120 router, firewall, access point, etc.). In some cases, the cable 121 modem function, i.e., the ability to access the Internet, is 122 integrated into a home gateway device or Embedded Multimedia Terminal 123 Adapter (eMTA). Once connected, the cable modem links the customer 124 to the HSI network and ultimately the broader Internet. 126 2.2. Cable Modem Termination System (CMTS) 128 A piece of hardware located in a cable operator's local network 129 (generally in a "headend", Section 2.10) that acts as the gateway to 130 the Internet for cable modems in a particular geographic area. A 131 simple way to think of the CMTS is as a router with interfaces on one 132 side leading to the Internet and interfaces on the other connecting 133 to Optical Nodes and then customers, in a so-called "last mile"" 134 network. 136 2.3. Cable Modem Termination System (CMTS) Port 138 Also referred to simply as a "port". A port is a physical interface 139 on a device used to connect cables in order to connect with other 140 devices for transferring information/data. An example of a physical 141 port is a CMTS port. A CMTS has both upstream and downstream network 142 interfaces to serve the local access network, which are referred to 143 as upstream or downstream ports. A port generally serves a 144 neighborhood of hundreds of homes. Over time, CMTS ports tend to 145 server fewer and fewer homes, as the network is segmented for 146 capacity growth purposes. Prior to DOCSIS version 3, a single CMTS 147 physical port was used for either transmitting or receiving data 148 downstream or upstream to a given neighborhood. With DOCSIS version 149 3, and the channel bonding feature, multiple CMTS physical ports can 150 be combined to create a virtual port. A CMTS is also briefly defined 151 in Section 2.6 of [RFC3083]. 153 2.4. Channel Bonding 155 A technique for combining multiple downstream and/or upstream 156 channels to increase customers' download and/or upload speeds, 157 respectively. Multiple channels from the Hybrid Fiber Coax (HFC, 158 Section 2.11) network can be bonded into a single virtual port 159 (called a bonded group), which acts as a large single channel or port 160 to provide increased speeds for customers. Channel bonding is a 161 feature of Data Over Cable Service Interface Specification (DOCSIS) 162 version 3, as described in the [CableLabs DOCSIS 3.0 MAC and Upper 163 Layer Protocols Interface Specification]. 165 2.5. Coaxial Cable (Coax) 167 A type of cable used by a cable operator to connect customer premise 168 equipment (CPE) -- such as TVs, cable modems (including eMTAs), and 169 Set Top Boxes -- to the HFC network. This cable may be used both 170 within the home as well as in segments of the last mile network 171 running to a home or customer premise location. There are many 172 grades of coaxial cable that are used for different purposes. 173 Different types of coaxial cable are used for different purposes on 174 the network. 176 2.6. Comcast High Speed Internet (HSI) 178 A service/product offered by Comcast for delivering Internet service 179 over a broadband connection. 181 2.7. Customer Premise Equipment (CPE) 183 Any device that resides at the customer's residence. 185 2.8. Data Over Cable Service Interface Specification (DOCSIS) 187 A reference standard developed by CableLabs that specifies how 188 components on cable networks need to be built to enable HSI service 189 over an HFC network, as noted in the [CableLabs DOCSIS 3.0 Cable 190 Modem to Customer Premise Equipment Interface Specification], 191 [CableLabs DOCSIS 3.0 Physical Layer Specification], [CableLabs 192 DOCSIS 3.0 MAC and Upper Layer Protocols Interface Specification], 193 [CableLabs DOCSIS 3.0 Security Specification], and [CableLabs DOCSIS 194 3.0 Operations Support System Interface Specification]. These 195 standards define the specifications for the cable modem and the CMTS 196 such that any DOCSIS certified cable modem will work on any DOCSIS- 197 certified CMTS, independent of the selected vendor. The 198 interoperability of cable modems and CMTSs allows customers to 199 purchase a DOCSIS-certified modem from a retail outlet and use it on 200 their cable-networked home. All DOCSIS-related standards are 201 available to the public at the CableLabs website, at 202 http://www.cablelabs.com. 204 2.9. Downstream 206 Description of the direction in which a signal travels, in this case 207 from the network to a user. Downstream traffic occurs when users are 208 downloading something from the Internet, such as watching a web-based 209 video, reading web pages, or downloading software updates. 211 2.10. Headend 213 A cable facility responsible for receiving TV signals for 214 distribution over the HFC network to the end customers. This 215 facility typically also houses one or more CMTSs. This is sometimes 216 also called a "hub". 218 2.11. Hybrid Fiber Coax (HFC) 220 A network architecture used primarily by cable companies, comprised 221 of fiber optic and coaxial cables that currently deliver Voice, 222 Video, and Internet services to customers, as defined in Section 1.2 223 of the [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface 224 Specification]. 226 2.12. Internet Protocol Detail Record (IPDR) 228 Standardized technology for monitoring and/or recording subscribers' 229 upstream and downstream Internet usage data based on their cable 230 modem. The data is collected from the CMTS and sent to a server for 231 further processing. Additional information is available at 232 http://www.ipdr.org, as well as [IPDR Standard] and [CableLabs DOCSIS 233 IPDR Support]. 235 2.13. Optical Node 237 A component of the HFC network generally located in customers' local 238 neighborhoods which is used to convert the optical signals sent over 239 fiber-optic cables to electrical signals that can be sent over 240 coaxial cable to customers' cable modems, or vice versa. A fiber 241 optic cable connects the Optical Node, through distribution hubs, to 242 the CMTS and coaxial cable connects the Optical Node to customers' 243 cable modems. 245 2.14. Provisioned Bandwidth 247 The peak speed associated with a tier of service purchased by a 248 customer. For example, a customer with a 50 Mbps downstream and 10 249 Mbps upstream speed tier would be said to be provisioned with 50 Mbps 250 of downstream bandwidth and 10 Mbps of upstream bandwidth. This is 251 often referred to as 50/10 service in industry parlance. 253 The Provisioned Bandwidth is the speed that a customer's modem is 254 configured (and the network is engineered) to deliver on a regular 255 basis (which is not the same as a "Committed Information Rate" or a 256 guaranteed rate). Internet speeds are generally a best efforts 257 service that are dependent on a number of variables, many of which 258 are outside the control of an Internet Service Provide (ISP). In 259 general, speeds do not typically exceed a customer's provisioned 260 speed. Comcast, however, invented a technology called "PowerBoost" 261 [PowerBoost Specification] that, for example, enables users to 262 experience brief boosts above their provisioned speeds while they 263 transfer large files over the Internet, by utilizing excess capacity 264 which may be available in the network at that time. 266 2.15. Quality of Service (QoS) 268 A set of techniques to manage network resources to ensure a level of 269 performance to specific data flows, as described in [RFC1633] and 270 [RFC2475]. One method for providing QoS to a network is by 271 differentiating the type of traffic by class or flow and assigning 272 priorities to each type. When the network becomes congested, the 273 data packets that are marked as having higher priority will have 274 higher likelihood of being serviced. 276 2.16. Upstream 278 Description of the direction in which a signal travels, in this case 279 from the user to the network. Upstream traffic occurs when users are 280 uploading something to the network, such as sending email, sending 281 files to another computer, or uploading photos to a digital photo 282 website. 284 3. Historical Overview 286 Comcast began the engineering project to develop a new congestion 287 management system in March 2008, the same month that Comcast hosted 288 the 71st meeting of the IETF in Philadelphia, PA, USA. On May 28, 289 2008, Comcast participated in an IETF Peer-to-Peer Infrastructure 290 Workshop [RFC5594], hosted by the Massachusetts Institute of 291 Technology (MIT) in Cambridge, MA, USA. 293 In order to participate in this workshop, interested attendees were 294 asked to submit a paper to a technical review team, which Comcast did 295 on May 9, 2008, in the [Comcast P2Pi Position Paper]. Comcast 296 subsequently attended and participated in this valuable workshop. 297 During the workshop, Comcast outlined the high-level design for a new 298 congestion management system [Comcast IETF P2P Infrastructure 299 Workshop Presentation] and solicited comments and other feedback from 300 attendees and other members of the Internet community (presentations 301 were also posted to the IETF's P2Pi mailing list). The congestion 302 management system outlined in that May 2008 workshop was later tested 303 in trial markets and is in essence what was then deployed by Comcast 304 later in 2008. 306 Following an August 2008 [FCC Memorandum and Opinion - August 2008] 307 regarding how Comcast managed congestion on its High-Speed Internet 308 ("HSI") network, Comcast disclosed to the FCC [FCC Network Management 309 Response - September 2008] and the public additional technical 310 details of the congestion management system that it intended to and 311 did implement by the end of 2008 [FCC Congestion Management 312 Deployment Completion Letter - January 2009], including the 313 thresholds involved in this new system. While the description of how 314 this system is deployed in the Comcast network is necessarily 315 specific to the various technologies and designs specific to that 316 network, a similar system could be deployed on virtually any large 317 scale ISP network or other IP network. 319 4. Summary 321 Comcast's HSI network has elements which are shared across many 322 subscribers. This means that Comcast's HSI customers share upstream 323 and downstream bandwidth with their neighbors. Although the 324 available bandwidth is substantial, so, too, is the demand. Thus, 325 when a relatively small number of customers in a neighborhood place 326 disproportionate demands on network resources, this can cause 327 congestion that degrades their neighbors' Internet experience. The 328 goal of Comcast's new congestion management system is to enable all 329 users of our network resources to access a "fair share" of that 330 bandwidth, in the interest of ensuring a high-quality online 331 experience for all of Comcast's HSI customers. 333 Importantly, the new approach is protocol-agnostic; that is, it does 334 not manage congestion by focusing on the use of the specific 335 protocols that place a disproportionate burden on network resources, 336 or any other protocols. Rather, the new approach focuses on managing 337 the traffic of those individuals who are using the most bandwidth at 338 times when network congestion threatens to degrade subscribers' 339 broadband experience and who are contributing disproportionately to 340 such congestion at those points in time. 342 Specific details about these practices, including relevant threshold 343 information, the type of equipment used, and other particulars, are 344 discussed at some length later in this document. At the outset, 345 however, we present a very high-level, simplified overview of how 346 these practices work. Despite all the detail provided further below, 347 the fundamentals of this approach can be summarized succinctly: 349 1. Software installed in the Comcast network continuously examines 350 aggregate traffic usage data for individual segments of Comcast's 351 HSI network. If overall upstream or downstream usage on a 352 particular segment of Comcast's HSI network reaches a pre- 353 determined level, the software moves on to step two. 355 2. At step two, the software examines bandwidth usage data for 356 subscribers in the affected network segment to determine which 357 subscribers are using a disproportionate share of the bandwidth. 358 If the software determines that a particular subscriber or 359 subscribers have been the source of high volumes of network 360 traffic during a recent period of minutes, traffic originating 361 from that subscriber or those subscribers temporarily will be 362 assigned a lower priority status. 364 3. During the time that a subscriber's traffic is assigned the lower 365 priority status, such traffic will not be delayed so long as the 366 network segment is not actually congested. If, however, the 367 network segment becomes congested, such traffic could be delayed. 369 4. The subscriber's traffic returns to normal priority status once 370 his or her bandwidth usage drops below a set threshold over a 371 particular time interval. 373 Comcast undertook considerable effort, over the course of many 374 months, to formulate our plans for this congestion management 375 approach, adjusting them, and subjecting them to real-world trials. 376 Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake 377 City, FL; East Orange, FL; and Colorado Springs, CO, between June and 378 September, 2008. This enabled us to validate the utility of the 379 general approach and collect substantial trial data to test multiple 380 variations and alternative formulations. 382 5. Implementation and Configuration 384 To understand exactly how these new congestion management practices 385 work, it is helpful to have a general understanding of how Comcast's 386 HSI network is designed. Comcast's HSI network is what is commonly 387 referred to as a hybrid fiber-coax network, with coaxial cable 388 connecting each subscriber's cable modem to an Optical Node, and 389 fiber optic cables connecting the Optical Node, through distribution 390 hubs, to the Cable Modem Termination System (CMTS), which is also 391 known as a "data node". The CMTSes are then connected to higher- 392 level routers, which in turn are connected to Comcast's Internet 393 backbone facilities. Today, Comcast has over 3,200 CMTSes deployed 394 throughout our network, serving over 15 million HSI subscribers. 396 Each CMTS has multiple "ports" that handle traffic coming into and 397 leaving the CMTS. In particular, each cable modem deployed on the 398 Comcast HSI network is connected to the CMTS through the ports on the 399 CMTS. These ports can be either "downstream" ports or "upstream" 400 ports, depending on whether they send information to cable modems 401 (downstream) or receive information from cable modems (upstream) 402 attached to the port. (Note that the term "port" as used here 403 generally contemplates single channels on a CMTS, but these 404 statements will apply to virtual channels, also known as "bonded 405 groups", in a DOCSIS 3.0 environment.) Even without channel bonding, 406 multiple channels are usually configured to come out each physical 407 port. Said another way, there is generally a mapping of multiple 408 channels to each physical port. 410 Currently, on average, approximately 275 cable modems share the same 411 downstream port and about 100 cable modems share the same upstream 412 port, however this is constantly changing (both numbers generally 413 become smaller over time). Both types of ports can experience 414 congestion that could degrade the broadband experience of our 415 subscribers and, unlike with the previous congestion management 416 practices, both upstream and downstream traffic are subject to 417 management in this new congestion management system. 419 To implement Comcast's new protocol-agnostic congestion management 420 practices, Comcast purchased new hardware and software that was 421 deployed near the Regional Network Routers ("RNRs") that are further 422 upstream in Comcast's network. This new hardware consists of 423 Internet Protocol Detail Record ("IPDR") servers, Congestion 424 Management servers, and PacketCable Multimedia ("PCMM") servers. 425 Further details about each of these pieces of equipment can be found 426 below, in Section 5.4. It is important to note here, however, that 427 even though the physical location of these servers is at the RNR, the 428 servers communicate with -- and manage individually -- multiple ports 429 on multiple CMTSes to effectuate the practices described in this 430 document. That is to say, bandwidth usage on one CMTS port will have 431 no effect on whether the congestion management practices described 432 herein are applied to a subscriber on a different CMTS port. 434 Figure 1 provides a simplified graphical depiction of the network 435 architecture just described: 437 Figure 1: Simplified Network Diagram Showing High-Level Comcast 438 Network and Servers Relevant to Congestion Management 440 ----------------- 441 ///// \\\\\ 442 | Comcast Internet Backbone | 443 \\\\\ ------ 444 ----------------// \\ 445 +------------+ / \ 446 | Congestion | | Internet | 447 | Management |<+++GigE++++ +---->| Backbone | 448 | Server | + | | Router | 449 +------------+ + | \ / 450 + Fiber \\ // 451 +------------+ + | ----- 452 | QoS | + | 453 | Server |<+++GigE++++ \/ 454 | | + ----- 455 +------------+ + // \\ 456 + / \ 457 +------------+ + | Regional | 458 | Statistics | +++++++>| Network | 459 | Collection |<+++GigE++++ | Router | 460 | Server | \ / 461 +------------+ +---Fiber------>\\ //<------Fiber----+ 462 | ----- | 463 | | 464 \/ \/ 465 ----- ----- 466 // \\ // \\ 467 / \ / \ 468 | Local | | Local | 469 | Market | | Market | 470 | Router | | Router | 471 \ / \ / 472 +--------->\\ //<------------+ \\ // 473 | ----- | ----- 474 | /\ | /\ 475 Fiber | Fiber | 476 | Fiber | Fiber 477 | | | | 478 \/ \/ \/ \/ 479 //----\\ //----\\ //----\\ //----\\ 480 | CMTS | | CMTS | | CMTS | | CMTS | 481 \\----// \\----// \\----// \\----// 482 /\ /\ /\ /\ 483 | | | | 484 Fiber Fiber Fiber Fiber 485 | | | | 486 \/ \/ \/ \/ 487 +---------+ +---------+ +---------+ +---------+ 488 | Optical | | Optical | | Optical | | Optical | 489 | Node | | Node | | Node | | Node | 490 +---------+ +---------+ +---------+ +---------+ 491 /\ /\ /\ /\ /\ /\ 492 || || ||______ || _____|| || 493 Coax Coax |__Coax| Coax |Coax__| Coax 494 || || || || || || 495 \/ \/ \/ \/ \/ \/ 496 +=======+ +=======+ +=======+ +=======+ +=======+ +=======+ 497 = Cable = = Cable = = Cable = = Cable = = Cable = = Cable = 498 = Modem = = Modem = = Modem = = Modem = = Modem = = Modem = 499 +=======+ +=======+ +=======+ +=======+ +=======+ +=======+ 501 ================================================================ 502 + Note: This diagram is a simplification of the actual network + 503 + and servers, which in actuality includes redundant network + 504 + links, redundant network devices, redundant servers, and + 505 + other details too complex to represent here. + 506 ================================================================ 508 Figure 1 510 Each Comcast HSI subscriber's cable modem has a "bootfile", which is 511 essentially a configuration file that contains certain pieces of 512 information about the subscriber's service to ensure that the service 513 functions properly. (Note: No personal information is included in 514 the bootfile; it only includes information about the service that the 515 subscriber has purchased.) For example, the bootfile contains 516 information about the maximum speed (what we refer to in this 517 document as the "provisioned bandwidth") that a particular modem can 518 achieve based on the tier (personal/residential, commercial, etc.) 519 the customer has purchased. Bootfiles are generally reset from time 520 to time to account for changes in the network and other updates, and 521 this is usually done through a command sent from the network and 522 without any effect on the subscriber. In preparation for the 523 transition to this new congestion management system, Comcast sent new 524 bootfiles to our HSI customers' cable modems that created two Quality 525 of Service ("QoS") levels for Internet traffic going to and from the 526 cable modem: (1) "Priority Best-Effort" traffic ("PBE"); and (2) 527 "Best-Effort" traffic ("BE"). As with previous changes to cable 528 modem bootfiles, the replacement of the old bootfile with the new 529 bootfile requires no active participation by Comcast customers. 531 Thereafter, all traffic going to or coming from cable modems on the 532 Comcast HSI network is designated as either PBE or BE. PBE is the 533 default status for all Internet traffic coming from or going to a 534 particular cable modem. Traffic is designated BE for a particular 535 cable modem only when both of two conditions are met: 537 o First, the usage level of a particular upstream or downstream port 538 of a CMTS, as measured over a particular period of time, must be 539 nearing the point where congestion could degrade users' 540 experience. We refer to this as the "Near Congestion State" and, 541 based on the technical trials we have conducted, we have 542 established a threshold, described in more detail below, for when 543 a particular CMTS port enters that state. 545 o Second, a particular subscriber must be making an extended, high 546 contribution to the bandwidth usage on the particular port, 547 relative to the service tier they purchased, as measured over a 548 particular period of time. We refer to this as the "Extended High 549 Consumption State" and, based on the technical trials we have 550 conducted, we have established a threshold, described in more 551 detail below, for when a particular user enters that state. 553 When, and only when, both conditions are met, a user's upstream or 554 downstream traffic (depending on which type of port is in the Near 555 Congestion State) is designated as BE. Then, to the extent that 556 actual congestion occurs, any delay resulting from the congestion 557 will affect BE traffic before it affects PBE traffic. 559 We now explain the foregoing in greater detail in the following 560 sections. 562 5.1. Thresholds For Determining When a CMTS Port Is in a Near 563 Congestion State 565 For a CMTS port to enter the Near Congestion State, traffic flowing 566 to or from that CMTS port must exceed a specified level (the "Port 567 Utilization Threshold") for a specific period of time (the "Port 568 Utilization Duration"). The Port Utilization Threshold on a CMTS 569 port is measured as a percentage of the total aggregate upstream or 570 downstream bandwidth for the particular port during the relevant 571 timeframe. The Port Utilization Duration on the CMTS is measured in 572 minutes. 574 Values for each of the thresholds to be used as part of this new 575 management technique have been tentatively established after an 576 extensive process of lab tests, simulations, technical trials, vendor 577 evaluations, customer feedback, and a third-party consulting 578 analysis. In the same way that specific anti-spam or other network 579 management practices are adjusted to address new issues that arise, 580 it is a near certainty that these values will change over time, as 581 Comcast gathers more data and performs additional analysis resulting 582 from wide-scale use of the new technique. Moreover, as with any 583 large network or software system, software bugs and/or unexpected 584 errors may arise, requiring software patches or other corrective 585 actions. As always, Comcast's decisions on these matters are driven 586 by the marketplace imperative that we deliver the best possible 587 experience to our HSI subscribers. 589 Given our experience so far, we have determined that a starting point 590 for the upstream Port Utilization Threshold should be 70 percent and 591 the downstream Port Utilization Threshold should be 80 percent. For 592 the Port Utilization Duration, we have determined that the starting 593 point should be approximately 15 minutes (although some technical 594 limitations in some newer CMTSes deployed on Comcast's network may 595 make this time period vary slightly). Thus, over any 15-minute 596 period, if an average of more than 70 percent of a port's upstream 597 bandwidth capacity or more than 80 percent of a port's downstream 598 bandwidth capacity is utilized, that port is determined to be in a 599 Near Congestion State. 601 Based on the trials conducted and operational experience to date, a 602 typical CMTS port on our HSI network is in a Near Congestion State 603 only for relatively small portions of the day, if at all, though 604 there is no way to forecast what will be the busiest time on a 605 particular port on a particular day. Moreover, the trial data and 606 operational experience indicate that, even when a particular port is 607 in a Near Congestion State, the instances where the network actually 608 becomes congested during the Port Utilization Duration are few, and 609 managed users whose traffic is delayed during those congested periods 610 perceive little, if any, effect, as discussed below. 612 5.2. Thresholds For Determining When a User Is in an Extended High 613 Consumption State and for Release from that Classification 615 Once a particular CMTS port is in a Near Congestion State, the 616 software examines whether any cable modems are consuming bandwidth 617 disproportionately. (Note: Although each cable modem is typically 618 assigned to a particular household, the software does not and cannot 619 actually identify individual users, the number of users sharing a 620 cable modem, or analyze particular users' traffic.) For purposes of 621 this document, we use "cable modem", "user", and "subscriber" 622 interchangeably to mean a subscriber account or user account and not 623 an individual person. For a user to enter an Extended High 624 Consumption State, he or she must consume greater than a certain 625 percentage of his or her provisioned upstream or downstream bandwidth 626 (the "User Consumption Threshold") for a specific length of time (the 627 "User Consumption Duration"). The User Consumption Threshold is 628 measured as a user's consumption of a particular percentage of his or 629 her total provisioned upstream or downstream bandwidth. That 630 bandwidth is the maximum speed that a particular modem can achieve 631 based on the tier (personal/residential, commercial, etc.) the 632 customer has purchased. For example, if a user buys a service with 633 speeds of 50 Mbps downstream and 10 Mbps upstream, then his or her 634 provisioned downstream speed is 50 Mbps and provisioned upstream 635 speed is 10 Mbps. It is also important to note that because the User 636 Consumption Threshold is a percentage of provisioned bandwidth for a 637 particular user account, and not a static value, users of higher 638 speed tiers have correspondingly higher User Consumption Thresholds. 639 Lastly, the User Consumption Duration is measured in minutes. 641 Following lab tests, simulations, technical trials, customer 642 feedback, vendor evaluations, and a third-party consulting analysis, 643 we have determined that the appropriate starting point for the User 644 Consumption Threshold is 70 percent of a subscriber's provisioned 645 upstream or downstream bandwidth, and that the appropriate starting 646 point for the User Consumption Duration is 15 minutes. That is, when 647 a subscriber uses an average of 70 percent or more of his or her 648 provisioned upstream or downstream bandwidth over a particular 15- 649 minute period, that user is then in an Extended High Consumption 650 State. Therefore, this is a consumption-based threshold and not a 651 peak-speed-based threshold. Thus, the Extended High Consumption 652 State is not tied to whether a user has bursted once or more above 653 this 70% threshold for a brief moment. Instead, it is consumption- 654 based, meaning that a certain bitrate must be exceeded over at least 655 the entire User Consumption Duration. 657 The User Consumption Thresholds have been set sufficiently high that 658 using the HSI connection for VoIP, web surfing, or most streaming 659 video cannot alone cause subscribers to our standard-level HSI 660 service to exceed the User Consumption Threshold. For example, while 661 one of Comcast's common HSI service tiers has a provisioned 662 downstream bandwidth of 15 Mbps today, streaming video (even some HD 663 video) from Hulu uses less than 2.5 Mbps, a Vonage or Skype VoIP call 664 uses less than 131 Kbps, and streaming music uses less than 128 kbps 665 (in this example, 70 percent of 15 Mbps is 10.5 Mbps). As noted 666 above, these values are subject to change as necessary in the same 667 way that specific anti-spam or other network management practices are 668 adjusted to address new issues that arise, or should unexpected 669 software bugs or other problems arise. 671 Based on data collected from the trial markets where the new 672 management practices are being tested, on average less than one-third 673 of one percent of subscribers have had their traffic priority status 674 changed to the BE state on any given day. For example, in Colorado 675 Springs, CO, the largest test market, on any given day in August 676 2008, an average of 22 users out of 6,016 total subscribers in the 677 trial had their traffic priority status changed to BE at some point 678 during the day. 680 A user's traffic is released from a BE state when the user's 681 bandwidth consumption drops below 50 percent of his or her 682 provisioned upstream or downstream bandwidth for a period of 683 approximately 15 minutes. These release criteria are intended to 684 minimize (and hopefully prevent) user QoS oscillation (hysteresis), 685 i.e., a situation in which a particular user could cycle repeatedly 686 between BE and PBE. NetForecast, Inc., an independent consultant 687 retained to provide analysis and recommendations regarding Comcast's 688 trials and related congestion management work, suggested this 689 approach, which has worked well in our ongoing trials and lab 690 testing. 692 Simply put, there are four steps to determining whether the traffic 693 associated with a particular cable modem is designated as PBE or BE: 695 1. Determine if the CMTS port is in a Near Congestion State. 697 2. If yes, determine whether any users are in an Extended High 698 Consumption State. 700 3. If yes, change those users' traffic to BE from PBE. If the 701 answer at either step one or step two is no, no action is taken. 703 4. If a user's traffic has been designated BE, check user 704 consumption at next interval. If user consumption has declined 705 below predetermined threshold, reassign the user's traffic as 706 PBE. If not, recheck at next interval. 708 Figure 2 graphically depicts how this congestion management process 709 works, using an example of a situation where upstream port 710 utilization may be reaching a Near Congestion State (the same 711 diagram, with different values in the appropriate places, could be 712 used to depict the management process for downstream ports, as well): 714 Figure 2: Upstream Congestion Management Decision Flowchart 716 /\ 717 / \ 718 +------------+ / \ +---------+ +---------+ 719 | Start | / \ | | / / 720 | Congestion | / \ | | / / 721 | Management +-->+ Question +--YES-->| Result |--THEN-->/ ACTION / 722 | Process | \ #1 / | #1 | / #1 / 723 | | \ / | | / / 724 +------------+ \ / +---------+ +---------+ 725 \ / | 726 \/ | 727 | THEN 728 | | 729 NO | 730 | \/ 731 | /\ 732 \/ / \ 733 +---------+ / \ 734 | | / \ 735 | | / \ 736 | Result |<-------------NO------------+ Question + 737 | #2 | \ #2 / 738 | | \ / 739 +---------+ \ / 740 \ / 741 \/ 742 | 743 | 744 YES 745 | 746 /\ | 747 / \ \/ 748 +---------+ / \ +---------+ 749 | | / \ | | 750 | | / \ THEN, AT | | 751 | Result |<--YES--+ Question + <---NEXT ANALYSIS------+ Result | 752 | #4 | \ #3 / POINT /\ | #3 | 753 | | \ / | | | 754 +---------+ \ / | +---------+ 755 \ / | 756 \/ | 757 | | 758 | | 759 +------------NO-------------+ 760 KEY TO FIGURES ABOVE: 761 Question #1: Is the CMTS Upstream Port Utilization at an average 762 of OVER 70% for OVER 15 minutes? 763 Result #1: CMTS marked in a Near Congestion State, indicating 764 congestion MAY occur soon. 765 Action #1: Search most recent analysis timeframe (approx. 15 mins.) 766 of IPDR usage data. 767 Question #2: Are any users consuming an average of OVER 70% of 768 provisioned upstream bandwidth for OVER 15 minutes? 769 Result #2: No action taken. 770 Result #3: Change user's upstream traffic from Priority Best Effort 771 (PBE) to Best Effort (BE). 772 Question #3: Is the user in Best Effort (BE) consuming an average 773 of LESS THAN 50% of provisioned upstream bandwidth? 774 Result #4: Change user's upstream traffic back to Priority Best 775 Effort (PBE) from Best Effort (BE). 777 Figure 2 779 5.3. Effect of BE Quality of Service on Users&apos Broadband Experience 781 When a CMTS port is in a Near Congestion State and a cable modem 782 connected to that port is in an Extended High Consumption State, that 783 cable modem's traffic is designated as BE. Depending upon the level 784 of congestion in the CMTS port, this designation may or may not 785 result in the user's traffic being delayed or, in extreme cases, 786 dropped before PBE traffic is dropped. This is because of the way 787 that the CMTS handles traffic. Specifically, CMTS ports have what is 788 commonly called a "scheduler" that puts all the packets coming from 789 or going to cable modems on that particular port in a queue and then 790 handles them in turn. A certain number of packets can be processed 791 by the scheduler in any given moment; for each time slot, PBE traffic 792 is given priority access to the available capacity, and BE traffic is 793 processed on a space-available basis. (It is important to note that 794 congestion can occur in any IP network, and, when it does, packets 795 can be delayed or dropped. As a result, applications and protocols 796 have been designed to deal with this reality. Our congestion 797 management system attempts to ensure that, in those rare cases where 798 packets may be dropped, BE packets are dropped before PBE packets are 799 dropped.) 801 A rough analogy would be to busses that empty and fill up at 802 incredibly fast speeds. As empty busses arrive at the figurative 803 "bus stop" -- every two milliseconds in this case -- they fill up 804 with as many packets as are waiting for "seats" on the bus, to the 805 limits of the bus' capacity. During non-congested periods, the bus 806 will usually have several empty seats, but, during congested periods, 807 the bus will fill up and packets will have to wait for the next bus. 808 It is in the congested periods that BE packets will be affected. If 809 there is no congestion, packets from a user in a BE state should have 810 little trouble getting on the bus when they arrive at the bus stop. 811 If, on the other hand, there is congestion in a particular instance, 812 the bus may become filled by packets in a PBE state before any BE 813 packets can get on. In that situation, the BE packets would have to 814 wait for the next bus that is not filled by PBE packets. In reality, 815 this all takes place in two-millisecond increments, so even if the 816 packets miss 50 "busses", the delay will only be about one-tenth of a 817 second. 819 During times of actual network congestion, when BE traffic might be 820 delayed, there are a variety of effects that could be experienced by 821 a user whose traffic is delayed, depending upon what applications he 822 or she is using. Typically, a user whose traffic is in a BE state 823 during actual congestion may find that a webpage loads sluggishly, a 824 peer-to-peer upload takes somewhat longer to complete, or a VoIP call 825 sounds choppy. Of course, the same thing could happen to the 826 customers on a port that is congested in the absence of any 827 congestion management; the difference here is that the effects of any 828 such delays are shifted toward those who have been placing the 829 greatest burden on the network, instead of being distributed randomly 830 among the users of that port without regard to their consumption 831 levels. As a matter of fact, our studies concluded that the 832 experience of the PBE subscribers improves when this congestion 833 management system is enabled. This conclusion is based on network 834 measurements, such as latency, and not the actual subscriber 835 experience. 837 NetForecast, Inc. explored the potential risk of a worst-case 838 scenario for users whose traffic is in a BE state: the possibility of 839 "bandwidth starvation" in the theoretical case where 100 percent of 840 the CMTS bandwidth is taken up by PBE traffic for an extended period 841 of time. In theory, such a condition could mean that a given user 842 whose traffic is designated BE would be unable to effectuate an 843 upload or download (as noted above, both are managed separately) for 844 some period of time. However, when these management techniques were 845 tested, first in company testbeds and then in our real-world trials 846 conducted in the five markets, such a theoretical condition did not 847 occur. In addition, our experience with the system as fully deployed 848 in our production network demonstrates that these management 849 practices have very modest real-world impacts. In addition, Comcast 850 did not receive a single customer complaint in any of the trial 851 markets that could be traced to this congestion management system, 852 despite having broadly publicized these trials. In our subsequent 853 national deployment into our production network, we still have yet to 854 find a specific complaint that can be traced back to the effected of 855 this congestion management system. 857 Comcast continues to monitor how user traffic is affected by these 858 new congestion management techniques and will make the adjustments 859 necessary to ensure that all Comcast HSI customers have a high- 860 quality Internet experience. 862 5.4. Equipment/Software Used and Location 864 The above-mentioned functions are carried out using three different 865 types of application servers, supplied by three different vendors. 866 As mentioned above, these servers are installed near Comcast's 867 regional network routers. The exact locations of these servers is 868 not particularly relevant to this document, as this does not change 869 the fact that they manage individual CMTS ports. 871 The first application server is an IPDR server, which collects 872 relevant cable modem volume usage information from the CMTS, such as 873 how many aggregate upstream or downstream bytes a subscriber uses 874 over a particular period of time. IPDR has been adopted as a 875 standard by many industry organizations and initiatives, such as 876 CableLabs, ATIS, ITU, and 3GPP, among others. 878 The second application server is the Congestion Management server, 879 which uses Simple Network Management Protocol ("SNMP") [RFC3410] to 880 measure CMTS port utilization and detect when a port is in a Near 881 Congestion State. When this happens, the Congestion Management 882 server then queries the relevant IPDR data for a list of cable modems 883 meeting the criteria set forth above for being in an Extended High 884 Consumption State. 886 If one or more users meet the criteria to be managed, then the 887 Congestion Management server notifies a third application server, the 888 PCMM application server, as to which users have been in an Extended 889 High Consumption State and whose traffic should be treated as BE. 890 The PCMM servers are responsible for signaling a given CMTS to set 891 the traffic for specific cable modems with a BE QoS, and for tracking 892 and managing the state of such CMTS actions. If no users meet the 893 criteria to be managed, no users will have their traffic managed. 895 Figure 3 graphically depicts the high-level management flows among 896 the congestion management components on Comcast's network, as 897 described above: 899 Figure 3: Simplified Diagram Showing High-Level Management Flows 900 Relevant to the System 902 +---------------+ +---------------+ 903 | Congestion | Instruct QoS Server | QoS | 904 | Management |******to Change QoS for****>| Server | 905 | Server | a Device | | 906 +----+---+------+ +-------+-------+ 907 /\ /\ * 908 | | * 909 X | Relay Selected * 910 | +---Statistics: Bytes---+ * 911 X Up/Down by Device | QoS Action: 912 | | Change from PBE 913 X +-------+-------+ to BE, or from 914 | | Statistics | BE to PBE 915 X | Collection | * 916 Periodic SNMP | Server | * 917 Requests to +---------------+ * 918 Check CMTS Port /\ * 919 Utilization | * 920 Levels Statistics Sent * 921 X Periodically From CMTS * 922 | | * 923 X +-------+-------+ * 924 | | CMTS | * 925 +-X-X-X-X-X-X-X-X->| in |<************ 926 | Headend | 927 +---------------+ 928 H /\ /\ H 929 H Internet H 930 H Traffic H 931 H to/from H 932 H User H 933 H \/ \/ H 934 /-------------------\ 935 /+-------------------+\ 936 / | User's Home | \ 937 / | | \ 938 | +---------+ | 939 | | Cable | | 940 | | Modem | | 941 | +---------+ | 942 | | 943 +-------------------+ 945 ================================================================= 946 = Notes: = 947 = 1-Statistics Collection Servers use IP Detail Records (IPDR). = 948 = 2-QoS Servers use PacketCable Multimedia (PCMM) = 949 = to set QoS gates on the CMTS. = 950 = 3-This figure is a simplificiation of the actual network and = 951 = servers, which included redundancies and other complexities = 952 = not necessary to depict the functional design. = 953 ================================================================= 955 Figure 3 957 6. Conclusion 959 Comcast's transition to this new protocol-agnostic congestion 960 management system began in October 2008, and Comcast completed the 961 transition on December 31, 2008. As described above, the new 962 approach does not manage congestion by focusing on managing the use 963 of specific protocols. Nor will this approach use TCP "reset 964 packets" [RFC3360]. Rather, the new system acts such that (1) during 965 periods when a CMTS port is in a Near Congestion State, (2) it 966 identifies the subscribers on that port who have consumed a 967 disproportionate amount of bandwidth over the preceding 15 minutes, 968 (3) it lowers the priority status of those subscribers' traffic to BE 969 status until those subscribers meet the release criteria, and (4) 970 during periods of congestion, delay BE traffic before PBE traffic is 971 delayed. Our trials and our subsequent network-wide deployment 972 indicate that this new congestion management system ensures a quality 973 online experience for all of our HSI customers. 975 7. Exceptional Network Utilization Considerations 977 This system was developed to cope with somewhat "normal" occurrences 978 of congestion that could occur on virtually any IP network. It 979 should also be noted, however, that such a system could also prove 980 particularly useful in the case of "exceptional network utilization" 981 events which existing network usage models do not or cannot 982 accurately predict. Some network operators refer to these 983 exceptional events as 'surges' in utilization, similar to sudden 984 surges in demand in electrical power grids, with which many people 985 may be familiar. 987 For example, in the case of a severe global pandemic, it may be 988 expected that large swaths of the population may need to work 989 remotely, via their Internet connection. In such a case, a largely 990 unprecedented level of utilization may occur. In such cases, it may 991 be helpful to have a flexible congestion management system that could 992 adapt to this situation and help allocate network resources while 993 additional capacity is being brought online or while a temporary 994 condition persists. 996 8. Security Considerations 998 There are no security considerations in this document. 1000 9. IANA Considerations 1002 There are no IANA considerations in this document. 1004 NOTE TO RFC EDITOR: PLEASE REMOVE THIS NULL SECTION PRIOR TO 1005 PUBLICATION. 1007 10. Acknowledgements 1009 The authors wish to acknowledge the hard work of the many people who 1010 helped to develop and/or review this document, as well as the people 1011 who helped deploy the system in such a short period of time. 1013 11. Change Log 1015 NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. 1017 o v03 - changed FCC doc reference to a more stable ref point, 1018 closing an open item. updated Mbps numbers to reflect current 1019 Internet speeds, closing an open item. also, closed out all 1020 editorial notes, closing an open item. additional tweaks following 1021 homework assignments send to co-authors. 1023 o v02 - lots of minor tweaks based on homework assignments that 1024 Jason handed out to co-authors, and closed out several open items 1026 o v01 - closed item on the open issue list - updated references 1028 o v00 - first version published 1030 12. Open Issues 1032 NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION. 1034 1. MAJOR: Add some content to the Security Considerations section 1036 2. MINOR: Make the ASCII art a bit smaller (vertically) 1038 3. MINOR: Double-check IPDR references - ensure most recent version 1039 of standard is referenced (incl. possibly DOCSIS 3.0 reference 1040 w/r/t IPDR) 1042 4. MINOR: Change the reference URL for PowerBoost to something more 1043 stable, perhaps from the USPTO 1045 5. MINOR: Before final update of document, update Mbps speeds to 1046 reflect then-current speeds. 1048 13. Informative References 1050 [CableLabs DOCSIS 3.0 Cable Modem to Customer Premise Equipment 1051 Interface Specification] 1052 CableLabs, "Data-Over-Cable Service Interface 1053 Specifications - DOCSIS 3.0 - Cable Modem to Customer 1054 Premise Equipment Interface Specification", DOCSIS 3.0 CM- 1055 SP-CMCIv3-I01-080320, March 2008, . 1059 [CableLabs DOCSIS 3.0 MAC and Upper Layer Protocols Interface 1060 Specification] 1061 CableLabs, "Data-Over-Cable Service Interface 1062 Specifications - DOCSIS 3.0 - MAC and Upper Layer 1063 Protocols Interface Specification", DOCSIS 3.0 CM-SP- 1064 MULPIv3.0-I11-091002, October 2009, . 1068 [CableLabs DOCSIS 3.0 Operations Support System Interface 1069 Specification] 1070 CableLabs, "Data-Over-Cable Service Interface 1071 Specifications - DOCSIS 3.0 - Operations Support System 1072 Interface Specification", DOCSIS 3.0 CM-SP-OSSIv3.0-I10- 1073 091002, October 2009, . 1076 [CableLabs DOCSIS 3.0 Physical Layer Specification] 1077 CableLabs, "Data-Over-Cable Service Interface 1078 Specifications - DOCSIS 3.0 - Physical Layer 1079 Specification", DOCSIS 3.0 CM-SP-PHYv3.0-I08-090121, 1080 January 2009, . 1083 [CableLabs DOCSIS 3.0 Security Specification] 1084 CableLabs, "Data-Over-Cable Service Interface 1085 Specifications - DOCSIS 3.0 - Security Specification", 1086 DOCSIS 3.0 CM-SP-SECv3.0-I11-091002, March 2008, . 1090 [CableLabs DOCSIS IPDR Support] 1091 Yassini, R., "Data-Over-Cable Service Interface 1092 Specifications - DOCSIS 2.0 - Operations Support System 1093 Interface Specification", DOCSIS 2.0 CM-SP-OSSIv2.0-C01- 1094 081104, November 2008, . 1097 [Comcast IETF P2P Infrastructure Workshop Presentation] 1098 Livingood, J. and R. Woundy, "Comcast's IETF P2P 1099 Infrastructure Workshop Presentation on May 28, 2008", FCC 1100 Filings Comcast Network Management Proceedings, May 2008, 1101 . 1104 [Comcast P2Pi Position Paper] 1105 Livingood, J. and R. Woundy, "Comcast's IETF P2P 1106 Infrastructure Workshop Position Paper", FCC 1107 Filings Comcast Network Management Proceedings, May 2008, 1108 . 1112 [FCC Congestion Management Deployment Completion Letter - January 1113 2009] 1114 Zachem, K., "Letter to the FCC Advising of Successful 1115 Deployment of Comcast's New Congestion Management System", 1116 FCC Filings Comcast Network Management Proceedings, 1117 January 2009, . 1120 [FCC Memorandum and Opinion - August 2008] 1121 Martin, K., Copps, M., Adelstein, J., Tate, D., and R. 1122 McDowell, "FCC Memorandum and Opinion Regarding Reasonable 1123 Network Management", File No. EB-08-IH-1518 WC Docket No. 1124 07-52, August 2008, . 1127 [FCC Network Management Response - September 2008] 1128 Zachem, K., "Letter to the FCC Regarding Comcast's Network 1129 Management Practices", FCC Filings Comcast Network 1130 Management Proceedings, September 2008, . 1133 [IPDR Standard] 1134 Cotton, S., Cockrell, B., Walls, P., and T. Givoly, 1135 "Network Data Management - Usage (NDM-U) For IP-Based 1136 Services Service Specification - Cable Labs DOCSIS 2.0 1137 SAMIS", IPDR Service Specifications NDM-U, November 2004, 1138 . 1141 [PowerBoost Specification] 1142 Comcast Cable Communications Management LLC, "Comcast 1143 PowerBoost Specification", Not Sure ValueHere, Month 2XXX, 1144 . 1149 [RFC1633] Braden, B., Clark, D., and S. Shenker, "Integrated 1150 Services in the Internet Architecture: an Overview", 1151 RFC 1633, June 1994. 1153 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1154 and W. Weiss, "An Architecture for Differentiated 1155 Services", RFC 2475, December 1998. 1157 [RFC3083] Woundy, R., "Baseline Privacy Interface Management 1158 Information Base for DOCSIS Compliant Cable Modems and 1159 Cable Modem Termination Systems", RFC 3083, March 2001. 1161 [RFC3360] Floyd, S., "Inappropriate TCP Resets Considered Harmful", 1162 BCP 60, RFC 3360, August 2002. 1164 [RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart, 1165 "Introduction and Applicability Statements for Internet- 1166 Standard Management Framework", RFC 3410, December 2002. 1168 [RFC5594] Peterson, J. and A. Cooper, "Report from the IETF Workshop 1169 on Peer-to-Peer (P2P) Infrastructure, May 28, 2008", 1170 RFC 5594, July 2009. 1172 Authors' Addresses 1174 Chris Bastian 1175 Comcast Cable Communications 1176 One Comcast Center 1177 1701 John F. Kennedy Boulevard 1178 Philadelphia, PA 19103 1179 US 1181 Email: chris_bastian@cable.comcast.com 1182 URI: http://www.comcast.com 1184 Tom Klieber 1185 Comcast Cable Communications 1186 One Comcast Center 1187 1800 Bishops Gate Drive 1188 Mount Laurel, NJ 08054 1189 US 1191 Email: tom_klieber@cable.comcast.com 1192 URI: http://www.comcast.com 1194 Jason Livingood (editor) 1195 Comcast Cable Communications 1196 One Comcast Center 1197 1701 John F. Kennedy Boulevard 1198 Philadelphia, PA 19103 1199 US 1201 Email: jason_livingood@cable.comcast.com 1202 URI: http://www.comcast.com 1204 Jim Mills 1205 Comcast Cable Communications 1206 One Comcast Center 1207 1800 Bishops Gate Drive 1208 Mount Laurel, NJ 08054 1209 US 1211 Email: jim_mills@cable.comcast.com 1212 URI: http://www.comcast.com 1213 Richard Woundy 1214 Comcast Cable Communications 1215 27 Industrial Avenue 1216 Chelmsford, MA 01824 1217 US 1219 Email: richard_woundy@cable.comcast.com 1220 URI: http://www.comcast.com