idnits 2.17.00 (12 Aug 2021) /tmp/idnits27170/draft-loughney-coach-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. == The page length should not exceed 58 lines per page, but there was 8 longer pages, the longest (page 2) being 60 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 9 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == Line 359 has weird spacing: '...imed to perta...' == Line 403 has weird spacing: '...>, and expir...' -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (23 June 2003) is 6900 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 4 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group John Loughney 3 INTERNET-DRAFT Nokia 4 Category: Informational Bernard Aboba 5 Microsoft 6 23 June 2003 8 A Comprehensive Approach to Quality (COACH) 10 This document is an Internet-Draft and is in full conformance with all 11 provisions of Section 10 of RFC 2026. 13 Internet-Drafts are working documents of the Internet Engineering Task 14 Force (IETF), its areas, and its working groups. Note that other groups 15 may also distribute working documents as Internet-Drafts. 17 Internet-Drafts are draft documents valid for a maximum of six months 18 and may be updated, replaced, or obsoleted by other documents at any 19 time. It is inappropriate to use Internet-Drafts as reference material 20 or to cite them other than as "work in progress." 22 The list of current Internet-Drafts can be accessed at 23 http://www.ietf.org/ietf/1id-abstracts.txt 25 The list of Internet-Draft Shadow Directories can be accessed at 26 http://www.ietf.org/shadow.html. 28 Copyright Notice 30 Copyright (C) The Internet Society (2003). All Rights Reserved. 32 Abstract 34 This document describes an approach to improving the quality of IETF 35 documents which is based on quality plans developed by Working Groups, 36 and approved by the Area Director. This document describes the 37 components of a quality plan, as well as a list of materials which need 38 to be compiled in order to assist Working Groups in developing their 39 plans. 41 1. Introduction 43 This document describes an approach to improving the quality of IETF 44 documents that centers around the development and execution of Working 45 Group quality plans. Since in the IETF Working Groups are responsible 46 for completing work, they are also accountable for the quality of that 47 work. If the quality of IETF documents is to improve, the Working 48 Groups must plan for, and carry out, that quality improvement. 50 This document describes how Working Group quality plans can be used to 51 assist in that effort - and what materials the IETF needs to put 52 together in order to assist Working Groups in developing their plans. 54 The Working Group quality plan is a public document no more than 5 pages 55 long that is developed in concert with the Working Group charter and is 56 made available on the IETF web site. The quality plan lays out the 57 challenges that the Working Group faces, and describes how it plans to 58 overcome those challenges in order to deliver documents of high quality. 59 The plan includes pointers to processes, tools, and templates that will 60 assist in the effort, as well as the plan for assessing the quality of 61 work at several checkpoints. 63 Working Group quality plans are made publicly available so as to 64 encourage sharing of best practices. Since each Working Group is 65 different, there is no "one size fits all" quality plan, although it is 66 expected that over time, as Working Groups share their plans with each 67 other, that best practices will be more widely adopted. 69 Typically, the quality plan will be revised when the Working Group 70 Charter or schedule changes significantly. For example, if the Working 71 Group has fallen badly behind in its schedule, then a revised Challenge 72 Assessment may be required in order to determine where the Working Group 73 resources have fallen short of those required to complete the work. 75 2. The quality plan 77 The Working Group quality plan includes (but is not limited to) the 78 following: 80 [a] Working Group Charter. The Working Group Charter is a summary of 81 what the Working Group plans to do, who will do it, and by when. 82 It includes a description of the documents that the Working Group 83 plans to produce, the individuals who have signed up to do the 84 work, and the schedule for when the work will be completed. Since 85 the Charter is a summary of the goals and objectives, as well as a 86 description of the team, it is the foundation for the Working Group 87 quality plan. 89 [b] Challenge Assessment. This part of the quality plan describes the 90 challenges that the Working Group faces in accomplishing the work 91 outlined in the charter. For example, this part of the plan 92 describes the technical issues that need to be solved in order to 93 complete the work, the estimated size of the documents that are to 94 be produced, the documents on which the work depends, and the areas 95 of knowledge, skill or experience in which the working group is 96 weak. In general, it is expected that the Challenge Assessment 97 will incorporate feedback from the Area Director and the IETF 98 community, so as to ensure that the Working Group has a realistic 99 view of the difficulty of the task. 101 [c] The Tools Plan. This part of the plan describes the tools that the 102 Working Group intends to use in order to manage the work. This 103 includes the Working Group mailing list and web site; issue 104 tracking and reporting tools; document revision control systems; 105 and document production and build environments. 107 [d] The Review plan. This part of the quality plan describes the 108 checkpoints at which the work will be reviewed, the review process 109 to be used, and how the reviewers will be recruited. 111 The elements of the Working Group quality plan are discussed in more 112 detail in the sections that follow. 114 2.1. Challenge Assessment 116 The Working Group Charter is the statement of what the Working Group is 117 attempting to achieve, as well as the resources at its disposal. The 118 Charter is essentially a promise to deliver (the goals) and a down 119 payment on that promise (the resources). 121 The question is whether the promise is realistic or not, and whether the 122 down payment is sufficient collateral against default. The purpose of 123 the Challenge Assessment is to provide a realistic assessment of what is 124 going to be necessary to deliver on the promises that the Working Group 125 has made - with high quality. 127 There are many reasons why Working Groups deliver poor quality 128 documents. However, in most cases where poor quality documents are 129 produced, it is unlikely that the plan was to deliver poor quality 130 documents all along. Rather, it is more likely that the level of 131 effort, skill and experience required to produce high quality documents 132 was beyond the capability of the Working Group participants, and that 133 poor quality was the result, not of bad intentions, but of insufficient 134 resources. 136 At the start of a project, it is easy to underestimate what will be 137 required to finish the job. This seems particularly true in the IETF, 138 where industries go from birth to maturity, and where individuals change 139 jobs, change their marital status, buy or build a house, go through a 140 mid-life crisis (or two), have children, or go trekking in Nepal, all in 141 the time it takes to bring an initial Internet-Draft to the point where 142 it is approved for publication as an RFC. 144 Often the group that is least able to estimate the resources required to 145 finish the proposed work are the core Working Group participants - if 146 only because a certain degree of naive optimism is required to start the 147 process in the first place. 149 The purpose of the Challenge Assessment is not to destroy that naive 150 optimism, but rather to augment it with an external assessment of 151 whether the resources identified in the Charter fall short of those 152 required to complete the work successfully. 154 An important element of the Challenge Assessment is a review of the 155 extent to which the charter goes beyond the current state of the art. 156 For example, are the security problems likely to be encountered ones 157 which can be solved by the application of existing security 158 technologies? Or are there unique aspects of the security problem for 159 which existing techniques are inapplicable? Does the Charter require 160 advances in research as well as engineering? 162 To a large degree, the Challenge Assessment focuses on risks outside the 163 core area of expertise of the Working Group. In general, IETF Working 164 Groups tend to do a reasonable job within their area, but weaknesses, 165 where they arise, are often the result of missing expertise in other 166 areas. For example, a Working Group in the Security Area will tend 167 handle security issues well, but may stumble over transport issues; an 168 Internet Area Working Group will tend to handle things correctly at the 169 IP layer, but may mishandle security; a Routing Area Working Group will 170 tend to properly handle routing, but may have difficulty with Operations 171 and Management. 173 The goal of the Challenge Assessment is to garner feedback from the IETF 174 community as a whole, in order to anticipate challenges that will need 175 to be overcome in order to complete the proposed work successfully. 176 Ideally, the Challenge Assessment will answer the question: "what does 177 the Working Group need to do in order to have a high propability of 178 completing the proposed work on schedule and with high quality?" 180 As a result of the Challenge Assessment, the Working Group charter and 181 quality plan may be modified so as to include additional resources in 182 certain areas. For example, if security issues are anticipated, then 183 security reviews may be added to the quality plan, and a security 184 advisor may be named as part of the Working Group Charter. If 185 production of large and complex documents is anticipated, then the Tools 186 plan will include use of sophisticated issue tracking and document 187 revision control systems, and the Working Group charter will include 188 additional editors with experience producing documents of that 189 complexity. 191 In addition to assessing the challenges that need to be overcome, the 192 Challenge Assessment will also identify key assumptions that if wrong, 193 would require the quality plan to be revised. For example, a key 194 assumption might be that transmission layer security (IPsec, TLS) will 195 be sufficient to address the security issues. If that assumption were 196 to be invalidated, then the significantly greater security resources 197 would be required - and the plan will need to be revised to take this 198 into account. 200 2.2. Tools Plan 202 The purpose of the Tools Plan is to identify the tools with which the 203 work will be carried out. 205 This includes: 207 [a] The technology and host for the Working Group mailing list. While 208 every Working Group has a mailing list, mailing list maintenance 209 has become increasingly complex as issues of spam control and 210 subscription spoofing have arisen. As a result, these and other 211 issues need to be taken into account in selecting how the mailing 212 list is run. 214 [b] The Working Group web site. Many Working Groups now use their own 215 web sites in order to track issues and provide access to work in 216 progress in formats not available on the IETF archive, such as XML 217 or HTML. In some cases, these web sites become integral parts of 218 the document production environment. The quality plan should 219 include a section on the plan for the web site. Ideally, the web 220 site will be hosted in a location that would not be disrupted by a 221 change in a participant's home address (home DSL or Cable links 222 have proven to be particularly vulnerable in this regard). 224 [c] The document production environment. Over the years, editors often 225 develop their own document production environments. These can vary 226 from the quick and dirty, suitable for the production of small 227 documents, to comprehensive environments that handle the building 228 of complex documents in multiple formats (e.g. text, HTML, XML, 229 etc.) as well as generation of difference files and propagation of 230 documents to the Working Group web site. Since each document is 231 different, there is no single right answer to the problem of 232 document production - but it does seem likely that Working Groups 233 can benefit by sharing sample build environments and templates and 234 documenting best practices. 236 [d] The revision control system. Not every Working Group will need to 237 maintain a revision control system, but where this is useful it is 238 probably a good idea to describe how revision control will be 239 handled. 241 [e] The issue tracking and reporting system. The use of issue tracking 242 tools has had a substantial impact on Working Group productivity 243 and document quality in a number of Working Groups. By focusing 244 effort on the identification and resolution of issues, tracking 245 tools encourage discussions that lead to document improvements. In 246 addition, tracking tools make it possible to avoid rehashing old 247 arguments and enable the quantitative measurement of progress 248 towards a goal. Given the increasing popularity of document 249 tracking tools, Working Groups are now experimenting with a number 250 of tools, and as part of the quality plan, it is helpful for the 251 chosen tools to be described along with the reasoning behind the 252 selection and pointers to relevant templates. As Issue Tracking 253 tools gain in popularity, use of reporting tools is also likely to 254 become more common. Reporting tools are essential for quality 255 improvement, since they supply the metrics by which progress can be 256 gauged. Use of quality metrics is not yet widespread with the 257 IETF, but where they are used, they should be discussed within the 258 quality plan. 260 2.3. Review Plan 262 The Review plan includes (but is not limited to) a description of the 263 plans for the following: 265 [a] Challenge Assessment review. This is plan for the challenge 266 assessment review, which is based on the Working Group Charter. 267 Ideally this plan should include reviewers from outside the area of 268 the Working Group. 270 [b] Work item review. In some cases, the Working Group may wish to 271 review documents before they become work items in order to identify 272 issues up front. If this is done, then the process by which these 273 reviews are conducted should be described. 275 [c] Midterm assessment. This is a review conducted when the document 276 is mid-way through its life. 278 [d] Late review. This is a review conducted late in the process, 279 within a few months of the start of Working Group last call. 281 As part of the review plan, the sources of reviewers should be 282 identified. This could be members of the relevant directorates, noted 283 experts, members of the IAB or IESG, volunteers in the process described 284 in [SIRS], etc. 286 3. Post Mortems 288 In order to increase the rate of quality improvement, it is important 289 for Working Groups to assess their quality plans after the fact. Prior 290 to dissolution, Working Groups are encouraged to produce a post mortem 291 no longer than three pages including the following sections: 293 [a] Quality assessment. This is an evaluation of the quality of working 294 group output, written by the Area Director. 296 [b] Challenge Assessment. This covers the unforseen challenges that the 297 Working Group encountered. 299 [c] Tools assessment. This is an evaluation of the tools that were used 300 and how well they worked. 302 [d] Reviews assessment. This is an assessment of the review process. 304 [e] Recommendations. This section includes recommendations to future 305 working groups looking to avoid similar problems. 307 Post mortems will be made publicly available along with the working 308 group quality plans. 310 4. Required materials 312 In order to assist Working Groups in putting together their quality 313 plans, the following materials would be helpful: 315 [a] An archive of sample document build environments and templates. 317 [b] A website covering issue tracking and reporting tools. 319 [c] A document on use of tracking tools for document management, 320 including metrics and reporting. 322 [d] A document on the review process, including Challenge Assessment. 324 [e] A website including sample quality plans and post-mortems. 326 5. Normative References 328 [SIRS] Carpenter, B. and D. Crocker, "Careful Additional Review 329 of Documents (CARD) by Senior IETF Reviewers (SIRS)", 330 draft-carpenter-solution-sirs-01.txt, Internet draft 331 (work in progress), June 2003. 333 Acknowledgments 335 The authors would like to acknowledge Margaret Wasserman and the members 336 of the IETF Problem Working Group for discussions relating to document 337 quality. 339 Authors' Addresses 341 John Loughney 342 Nokia 343 It204merenkatu 11-13 344 00180 Helsinki 345 Finland 347 Email: john.loughney@nokia.com 349 Bernard Aboba 350 Microsoft Corporation 351 One Microsoft Way 352 Redmond, WA 98052 354 EMail: bernarda@microsoft.com 356 Intellectual Property Statement 358 The IETF takes no position regarding the validity or scope of any 359 intellectual property or other rights that might be claimed to pertain 360 to the implementation or use of the technology described in this 361 document or the extent to which any license under such rights might or 362 might not be available; neither does it represent that it has made any 363 effort to identify any such rights. Information on the IETF's 364 procedures with respect to rights in standards-track and standards- 365 related documentation can be found in BCP-11. Copies of claims of 366 rights made available for publication and any assurances of licenses to 367 be made available, or the result of an attempt made to obtain a general 368 license or permission for the use of such proprietary rights by 369 implementors or users of this specification can be obtained from the 370 IETF Secretariat. 372 The IETF invites any interested party to bring to its attention any 373 copyrights, patents or patent applications, or other proprietary rights 374 which may cover technology that may be required to practice this 375 standard. Please address the information to the IETF Executive 376 Director. 378 Full Copyright Statement 380 Copyright (C) The Internet Society (2003). All Rights Reserved. 381 This document and translations of it may be copied and furnished to 382 others, and derivative works that comment on or otherwise explain it or 383 assist in its implementation may be prepared, copied, published and 384 distributed, in whole or in part, without restriction of any kind, 385 provided that the above copyright notice and this paragraph are included 386 on all such copies and derivative works. However, this document itself 387 may not be modified in any way, such as by removing the copyright notice 388 or references to the Internet Society or other Internet organizations, 389 except as needed for the purpose of developing Internet standards in 390 which case the procedures for copyrights defined in the Internet 391 Standards process must be followed, or as required to translate it into 392 languages other than English. The limited permissions granted above are 393 perpetual and will not be revoked by the Internet Society or its 394 successors or assigns. This document and the information contained 395 herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE 396 INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR 397 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 398 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 399 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 401 Expiration Date 403 This memo is filed as , and expires 404 December 22, 2003.