idnits 2.17.00 (12 Aug 2021) /tmp/idnits53752/draft-bormann-t2trg-stp-03.txt: -(3): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. == It seems as if not all pages are separated by form feeds - found 10 form feeds but 16 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (8 April 2020) is 766 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Bormann 3 Internet-Draft Universität Bremen TZI 4 Intended status: Informational K. Hartke 5 Expires: 10 October 2020 Ericsson 6 8 April 2020 8 The Series Transfer Pattern (STP) 9 draft-bormann-t2trg-stp-03 11 Abstract 13 Many applications make use of Series of data items, i.e., an array of 14 data items where new items can be added over time. Where such Series 15 are to be made available using REST protocols such as CoAP or HTTP, 16 the Series has to be mapped into a structure of one or more resources 17 and a protocol for a client to obtain the Series and to learn about 18 new items. 20 Various protocols have been standardized that make Series-shaped data 21 available, with rather different properties and objectives. The 22 present document is an attempt to extract a common underlying pattern 23 and to define media types and an access scheme that can be used right 24 away for further protocols that provide Series-shaped data. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on 10 October 2020. 43 Copyright Notice 45 Copyright (c) 2020 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 50 license-info) in effect on the date of publication of this document. 51 Please review these documents carefully, as they describe your rights 52 and restrictions with respect to this document. Code Components 53 extracted from this document must include Simplified BSD License text 54 as described in Section 4.e of the Trust Legal Provisions and are 55 provided without warranty as described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 2. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 3. A REST Series Transfer Pattern (STP) . . . . . . . . . . . . 4 62 3.1. Basic collections . . . . . . . . . . . . . . . . . . . . 4 63 3.2. Pagination and Observing linked lists . . . . . . . . . . 5 64 3.3. The "cursor" pattern . . . . . . . . . . . . . . . . . . 7 65 4. IANA considerations . . . . . . . . . . . . . . . . . . . . . 8 66 5. Security considerations . . . . . . . . . . . . . . . . . . . 9 67 6. Informative References . . . . . . . . . . . . . . . . . . . 9 68 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 10 69 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 71 1. Introduction 73 (TO DO: Insert an extended form of the abstract first here, expanding 74 the reference to [RFC7230] and [RFC7252] in the process.) 76 Examples for protocols that provide Series-shaped data are: 78 * The Atom Syndication Format [RFC4287] defines _feeds_ as Series of 79 _entries_ (links plus some metadata, which can often be much of 80 the content of an entry), where a feed is represented by a 81 collection resource that contains just a small number of the most 82 recent entries. By polling a feed, a client can contain a fresh 83 view of the Series, with a focus on recent items. If the client 84 does not poll often enough, it will _miss_ items. 86 * Messaging protocols such as XMPP or SIMPLE transfer series of what 87 is often called "Instant Messages". A publish/subscribe mechanism 88 allows a client to select sequences of messages that it is 89 interested in. 91 * Mail servers that provide interactive access to stored messages 92 present a Series to their clients. Obviously, loss of messages is 93 frowned upon. 95 * CoAP Observe allows a client to observe a resource as it changes; 96 the client can collect the changes into a Series. Observe is 97 focused on eventual consistency: a fresher data items simply 98 overwrites an older one. The present document uses the observe 99 pattern to build a more general Series Transfer Pattern. 101 * Syslog is an interesting case of a Series Transfer. 103 * [RFC8641], [I-D.voit-netmod-yang-notifications2], [RFC8639], 104 [I-D.ietf-netconf-notification-messages], [RFC8650]. 106 * An RTP stream can be viewed as an (somewhat extreme) case of a 107 Series; new items are just sent inside separate UDP packets. A 108 sequence number allows to detect (but not normally ask for 109 retransmission of) missing items. A timestamp as well as source 110 data (SSRC, CSRC) provide further common metadata that aid in the 111 processing of the Series items. 113 * An example of an ad-hoc design of a series transfer protocol is 114 [I-D.ietf-netconf-udp-pub-channel]. 116 * Server-sent events [sse] are a somewhat bizarre version of a 117 series transfer protocol. 119 * The Interface for Metadata Access Points (IF-MAP) specified by the 120 Trusted Computing Group and emerging derivatives of that protocol 121 create a series of updates to a graph representation of related 122 network-related security information. The requests created by IF- 123 MAP clients are bundled operations of updates to a MAP Graph, 124 which compose a Series Transfer Pattern of bundled atomic 125 operations that ensure the integrity of the MAP Graph. [Henk 126 Birkholz] 128 * netflow/IPFIX was defined to transfer a series of data items about 129 flows. Information about PDU flows accounted by network 130 interfaces of endpoints is emitted in a series of counter bundles 131 via the IPFIX protocol. Only a series of these continuous Flow 132 Records creates a meaningful bigger picture about the current 133 traffic in the network topology of an administrative domain. 134 Depending on the characteristics measured, loss of a Flow Record 135 can range from harmless to missing the only vital counter 136 measurement. [Henk Birkholz] 138 * TO DO: Add more items. 140 [I-D.birkholz-yang-push-coap-problemstatement] is a problem statement 141 that will require the design of another scheme to transfer Series- 142 shaped data. 144 2. Objectives 146 Series transfer applications may have rather different objectives. 148 * The completeness of the Series transfer may be of utmost 149 importance (e.g., if each item represents a sale), it may be 150 desirable but can be jettisoned in an overload situation, or it 151 may just be a likely outcome with a very active client (e.g., with 152 Atom). Note that there is never a way to _guarantee_ completeness 153 unless all of the rate and size of incoming new items, the 154 transfer capacity available, and the processing capabilities of 155 the client are controlled; however, system designs may want to 156 give the illusion of "reliability". 158 * Minimizing the latency of the transfer may be important, as may be 159 limiting it below a defined maximum (note that these are different 160 objectives). The latter can be supported in a polling system by 161 polling at least as often as that maximum latency; this may be 162 considered inefficient and "push" mechanisms may be developed. 163 Mail environments have developed "push" services to enable 164 minimizing the latency. Where latency requirements go below the 165 time that might be needed for an end-to-end retransmission, error 166 concealment may provide an acceptable user experience (e.g., in 167 RTP). 169 In general, minimizing latency and ensuring completeness are 170 competing objectives. 172 Series transfer environments sometimes centralize information 173 distribution functions, leading to "broker" architectures (often 174 combined with the "publish/subscribe" pattern). With brokers, Series 175 publishers may use an entirely different interface to the brokers 176 from that used by the receiving clients, or the interfaces can be 177 designed so they are similar for all the forwarding steps. 179 3. A REST Series Transfer Pattern (STP) 181 3.1. Basic collections 183 A series of items can be represented by a single collection resource: 185 _____________ 186 | | 187 | item 11 | 188 |_____________| 189 | | 190 | item 10 | 191 |_____________| 192 | . | 193 | . | 194 | . | 195 |_____________| 196 | | 197 | item 1 | 198 |_____________| 200 Figure 1: A collection of items 202 While this is adequate in many cases, it has a number of limitations: 204 * Each retrieval fetches the entire collection 206 - As long as the collection does not change, this can be 207 mitigated with ETags (Section 5.10.6 of[RFC7252], Section 2.3 208 of [RFC7232]). 210 * When the collection becomes too large, the server has to prune 211 older items. These then no longer can be retrieved, and there is 212 even no way for the server to indicate that there used to be older 213 items. 215 3.2. Pagination and Observing linked lists 217 In the Browser Web, it is usual to provide _Pagination_ for 218 collection resources that can grow large (e.g., search results): 220 _____________ _____________ _____________ 221 | | | | | | 222 | item 11 | +--->| item 9 | +--->| item 2 | 223 |_____________| | |_____________| | |_____________| 224 | | | | | | | 225 | item 10 | | | item 8 | ::::: | item 1 | 226 |_____________| | |_____________| |_____________| 227 | | | | | | page M 228 | link -----------+ | link -----------+ 229 |_____________| |_____________| 231 Figure 2: A paginated collection of items 233 Without modification, this does not work well for resources that 234 actually change by themselves: Once a new page needs to be added, 235 what previously was page 1 now becomes page 2. Obviously, the naming 236 of pages better remains unchanged with new pages added a the front. 238 _____________ _____________ _____________ 239 | | | | | | 240 | item 11 | +--->| item 9 | +--->| item 2 | 241 |_____________| | |_____________| | |_____________| 242 | | | | | | | 243 | item 10 | | | item 8 | ::::: | item 1 | 244 |_____________| | |_____________| |_____________| 245 | link -----------+ | link -----------+ 246 |_____________| |_____________| 248 Figure 3: Pagination with stable names 250 However, now the client has no idea what initial page to request to 251 get the freshest items and the head of the list. It is easy to add a 252 link to the freshest page: 254 _____________ _____________ _____________ 255 | | | | | | 256 | link --------------->| item 11 | +--->| item 2 | 257 |_____________| |_____________| | |_____________| 258 head | | | | 259 | item 10 | ::::: | item 1 | 260 |_____________| |_____________| 261 | link -----------+ 262 |_____________| 263 page M 265 Figure 4: Pagination with stable names 267 The head of the linked list can now be simply observed; the addition 268 of pages will then be notified to the observer. 270 As usual in series transfer, the following considerations remain: 272 * When can the server decide to no longer retain older items? 274 - There may be a desire for an observer to be able to catch all 275 items in the series. 277 o How does the server know who are the observers? E.g., what 278 to do with newly joining observers? 280 o How does an observer signal that it has caught up (to a 281 specific item)? 283 * What to do when the decision to remove items from the list cannot 284 be made and there is no room for new items? 286 The link head can also include items that have so far not been added 287 to pages; this can be used to fill up pages evenly without them ever 288 changing. Obviously, the best number of items to prenotify in this 289 way as well as the best time to open a new page are different for 290 different applications. 292 3.3. The "cursor" pattern 294 A GET on a resource representing a Series may return a collection 295 item that contains the following pieces of information 297 * An array of Series items, either as an array of media-typed 298 objects in a suitable representation format (e.g., CBOR, MIME) or 299 by using an array-like media type (e.g., SenML). 301 - Items may be full items or limit themselves to some metadata 302 and a link; the client can then follow that link if it is 303 interested in the data (possibly basing that decision on the 304 metadata and/or a measure of load). 306 * A "cursor" that can then be used as a parameter in further GET 307 requests (see below) in order to receive only newer items than 308 those received with this transfer. 310 * A "more bit" that indicates whether such further items already 311 exist but could not be returned in the present response. 313 In Figure 5, the cursor is implemented as a URI that can be used as a 314 link to the next page. 316 _____________ _____________ _____________ 317 | | | | | | 318 | item 10 |<---+ | item 1 |<--------------- link | 319 |_____________| | |_____________| |_____________| 320 | | | | tail 321 | item 11 | ::::: | item 2 | 322 |_____________| |_____________| 323 page M | | | 324 +----------- link | 325 |_____________| 327 Figure 5: Cursor pattern pictured as pagination 329 A GET may be enhanced with additional parameters (possibly turning it 330 into a FETCH): 332 * The cursor. 334 * A "wait bit" that indicates whether a (possibly empty) reply 335 should be given right away or the server should wait for new items 336 to become available. (To avoid the equivalence of the "silly 337 window syndrome", the wait bit may be enhanced by a minimum number 338 of items and a timeout after which even a smaller number is made 339 available.) In effect, this requests a form of "long polling"; 340 see [RFC6202] for some considerations for this in HTTP. 342 A server may implement a form of custody transfer by interpreting the 343 cursor as an acknowledgement that the client has received all data up 344 to the cursor. This is not necessarily acting as an unsafe request 345 ("destructive GET"), as other clients may be active that have not yet 346 received all these data. To implement a full custody semantics, the 347 server needs to be aware of all the clients that expect a full Series 348 Transfer (a classical group management problem). 350 (Explain how Observe can help. Can it?) 352 4. IANA considerations 354 This memo registers a number of media types: TO DO. 356 * A media type for FETCH selectors (Section 3): 358 - An alternative way to encode this information into the URI of a 359 GET should also be available. 361 * A Series media type as alluded to in Section 3. 363 5. Security considerations 365 TO DO 367 6. Informative References 369 [I-D.birkholz-yang-push-coap-problemstatement] 370 Birkholz, H., Zhou, T., Liu, X., and E. Voit, "YANG Push 371 Operations for CoMI", Work in Progress, Internet-Draft, 372 draft-birkholz-yang-push-coap-problemstatement-00, 18 373 October 2017, . 376 [I-D.ietf-netconf-notification-messages] 377 Voit, E., Jenkins, T., Birkholz, H., Bierman, A., and A. 378 Clemm, "Notification Message Headers and Bundles", Work in 379 Progress, Internet-Draft, draft-ietf-netconf-notification- 380 messages-08, 17 November 2019, . 384 [I-D.ietf-netconf-udp-pub-channel] 385 Zheng, G., Zhou, T., and A. Clemm, "UDP based Publication 386 Channel for Streaming Telemetry", Work in Progress, 387 Internet-Draft, draft-ietf-netconf-udp-pub-channel-05, 11 388 March 2019, . 391 [I-D.voit-netmod-yang-notifications2] 392 Voit, E., Bierman, A., Clemm, A., and T. Jenkins, "YANG 393 Notification Headers and Bundles", Work in Progress, 394 Internet-Draft, draft-voit-netmod-yang-notifications2-00, 395 24 February 2017, . 398 [RFC4287] Nottingham, M., Ed. and R. Sayre, Ed., "The Atom 399 Syndication Format", RFC 4287, DOI 10.17487/RFC4287, 400 December 2005, . 402 [RFC6202] Loreto, S., Saint-Andre, P., Salsano, S., and G. Wilkins, 403 "Known Issues and Best Practices for the Use of Long 404 Polling and Streaming in Bidirectional HTTP", RFC 6202, 405 DOI 10.17487/RFC6202, April 2011, 406 . 408 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 409 Protocol (HTTP/1.1): Message Syntax and Routing", 410 RFC 7230, DOI 10.17487/RFC7230, June 2014, 411 . 413 [RFC7232] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 414 Protocol (HTTP/1.1): Conditional Requests", RFC 7232, 415 DOI 10.17487/RFC7232, June 2014, 416 . 418 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 419 Application Protocol (CoAP)", RFC 7252, 420 DOI 10.17487/RFC7252, June 2014, 421 . 423 [RFC8639] Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard, 424 E., and A. Tripathy, "Subscription to YANG Notifications", 425 RFC 8639, DOI 10.17487/RFC8639, September 2019, 426 . 428 [RFC8641] Clemm, A. and E. Voit, "Subscription to YANG Notifications 429 for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641, 430 September 2019, . 432 [RFC8650] Voit, E., Rahman, R., Nilsen-Nygaard, E., Clemm, A., and 433 A. Bierman, "Dynamic Subscription to YANG Events and 434 Datastores over RESTCONF", RFC 8650, DOI 10.17487/RFC8650, 435 November 2019, . 437 [sse] WHATWG, "HTML Living Standard -- 9.2 Server-sent events", 438 n.d., . 441 Acknowledgements 443 The need for a Series Transfer Pattern has been made clear by a 444 number of people that contribute to the IRTF Thing-to-Thing Research 445 Group (T2TRG), e.g. Matthias Kovatsch and Henk Birkholz (both of 446 whom also provided feedback on an early draft). Henk also 447 contributed further examples for the use of Series Transfers in 448 protocols. 450 Authors' Addresses 452 Carsten Bormann 453 Universität Bremen TZI 454 Postfach 330440 455 D-28359 Bremen 456 Germany 457 Phone: +49-421-218-63921 458 Email: cabo@tzi.org 460 Klaus Hartke 461 Ericsson 462 Torshamnsgatan 23 463 SE-16483 Stockholm 464 Sweden 466 Email: klaus.hartke@ericsson.com