idnits 2.17.00 (12 Aug 2021) /tmp/idnits9969/draft-ietf-quic-qlog-main-schema-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 30 instances of too long lines in the document, the longest one being 329 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (10 June 2021) is 344 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-01) exists of draft-ietf-quic-qlog-h3-events-00 == Outdated reference: A later version (-01) exists of draft-ietf-quic-qlog-quic-events-00 Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 QUIC R. Marx, Ed. 3 Internet-Draft KU Leuven 4 Intended status: Standards Track L. Niccolini, Ed. 5 Expires: 12 December 2021 Facebook 6 M. Seemann, Ed. 7 Protocol Labs 8 10 June 2021 10 Main logging schema for qlog 11 draft-ietf-quic-qlog-main-schema-00 13 Abstract 15 This document describes a high-level schema for a standardized 16 logging format called qlog. This format allows easy sharing of data 17 and the creation of reusable visualization and debugging tools. The 18 high-level schema in this document is intended to be protocol- 19 agnostic. Separate documents specify how the format should be used 20 for specific protocol data. The schema is also format-agnostic, and 21 can be represented in for example JSON, csv or protobuf. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on 12 December 2021. 40 Copyright Notice 42 Copyright (c) 2021 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 47 license-info) in effect on the date of publication of this document. 48 Please review these documents carefully, as they describe your rights 49 and restrictions with respect to this document. Code Components 50 extracted from this document must include Simplified BSD License text 51 as described in Section 4.e of the Trust Legal Provisions and are 52 provided without warranty as described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 1.1. Notational Conventions . . . . . . . . . . . . . . . . . 4 58 2. Design goals . . . . . . . . . . . . . . . . . . . . . . . . 5 59 3. The high level qlog schema . . . . . . . . . . . . . . . . . 6 60 3.1. summary . . . . . . . . . . . . . . . . . . . . . . . . . 7 61 3.2. traces . . . . . . . . . . . . . . . . . . . . . . . . . 8 62 3.3. Individual Trace containers . . . . . . . . . . . . . . . 9 63 3.3.1. configuration . . . . . . . . . . . . . . . . . . . . 10 64 3.3.2. vantage_point . . . . . . . . . . . . . . . . . . . . 12 65 3.4. Field name semantics . . . . . . . . . . . . . . . . . . 14 66 3.4.1. timestamps . . . . . . . . . . . . . . . . . . . . . 15 67 3.4.2. category and event . . . . . . . . . . . . . . . . . 17 68 3.4.3. data . . . . . . . . . . . . . . . . . . . . . . . . 18 69 3.4.4. protocol_type . . . . . . . . . . . . . . . . . . . . 19 70 3.4.5. triggers . . . . . . . . . . . . . . . . . . . . . . 19 71 3.4.6. group_id . . . . . . . . . . . . . . . . . . . . . . 20 72 3.4.7. common_fields . . . . . . . . . . . . . . . . . . . . 21 73 4. Guidelines for event definition documents . . . . . . . . . . 23 74 4.1. Event design guidelines . . . . . . . . . . . . . . . . . 23 75 4.2. Event importance indicators . . . . . . . . . . . . . . . 24 76 4.3. Custom fields . . . . . . . . . . . . . . . . . . . . . . 25 77 5. Generic events and data classes . . . . . . . . . . . . . . . 25 78 5.1. Raw packet and frame information . . . . . . . . . . . . 26 79 5.2. Generic events . . . . . . . . . . . . . . . . . . . . . 27 80 5.2.1. error . . . . . . . . . . . . . . . . . . . . . . . . 27 81 5.2.2. warning . . . . . . . . . . . . . . . . . . . . . . . 27 82 5.2.3. info . . . . . . . . . . . . . . . . . . . . . . . . 27 83 5.2.4. debug . . . . . . . . . . . . . . . . . . . . . . . . 28 84 5.2.5. verbose . . . . . . . . . . . . . . . . . . . . . . . 28 85 5.3. Simulation events . . . . . . . . . . . . . . . . . . . . 28 86 5.3.1. scenario . . . . . . . . . . . . . . . . . . . . . . 29 87 5.3.2. marker . . . . . . . . . . . . . . . . . . . . . . . 29 88 6. Serializing qlog . . . . . . . . . . . . . . . . . . . . . . 29 89 6.1. qlog to JSON mapping . . . . . . . . . . . . . . . . . . 30 90 6.1.1. numbers . . . . . . . . . . . . . . . . . . . . . . . 30 91 6.1.2. bytes . . . . . . . . . . . . . . . . . . . . . . . . 31 92 6.1.3. Summarizing table . . . . . . . . . . . . . . . . . . 33 93 6.1.4. Other JSON specifics . . . . . . . . . . . . . . . . 33 94 6.2. qlog to NDJSON mapping . . . . . . . . . . . . . . . . . 34 95 6.2.1. Supporting NDJSON in tooling . . . . . . . . . . . . 35 96 6.3. Other optimizated formatting options . . . . . . . . . . 35 97 6.3.1. Data structure optimizations . . . . . . . . . . . . 36 98 6.3.2. Compression . . . . . . . . . . . . . . . . . . . . . 37 99 6.3.3. Binary formats . . . . . . . . . . . . . . . . . . . 38 100 6.3.4. Overview and summary . . . . . . . . . . . . . . . . 39 101 6.4. Conversion between formats . . . . . . . . . . . . . . . 40 102 7. Methods of access and generation . . . . . . . . . . . . . . 41 103 7.1. Set file output destination via an environment 104 variable . . . . . . . . . . . . . . . . . . . . . . . . 41 105 7.2. Access logs via a well-known endpoint . . . . . . . . . . 42 106 8. Tooling requirements . . . . . . . . . . . . . . . . . . . . 43 107 9. Security and privacy considerations . . . . . . . . . . . . . 44 108 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 44 109 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 44 110 11.1. Normative References . . . . . . . . . . . . . . . . . . 44 111 11.2. Informative References . . . . . . . . . . . . . . . . . 44 112 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 45 113 A.1. Since draft-marx-qlog-main-schema-draft-02: . . . . . . . 45 114 A.2. Since draft-marx-qlog-main-schema-01: . . . . . . . . . . 45 115 A.3. Since draft-marx-qlog-main-schema-00: . . . . . . . . . . 46 116 Appendix B. Design Variations . . . . . . . . . . . . . . . . . 46 117 Appendix C. Acknowledgements . . . . . . . . . . . . . . . . . . 46 118 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 47 120 1. Introduction 122 There is currently a lack of an easily usable, standardized endpoint 123 logging format. Especially for the use case of debugging and 124 evaluating modern Web protocols and their performance, it is often 125 difficult to obtain structured logs that provide adequate information 126 for tasks like problem root cause analysis. 128 This document aims to provide a high-level schema and harness that 129 describes the general layout of an easily usable, shareable, 130 aggregatable and structured logging format. This high-level schema 131 is protocol agnostic, with logging entries for specific protocols and 132 use cases being defined in other documents (see for example 133 [QLOG-QUIC] for QUIC and [QLOG-H3] for HTTP/3 and QPACK-related event 134 definitions). 136 The goal of this high-level schema is to provide amenities and 137 default characteristics that each logging file should contain (or 138 should be able to contain), such that generic and reusable toolsets 139 can be created that can deal with logs from a variety of different 140 protocols and use cases. 142 As such, this document contains concepts such as versioning, metadata 143 inclusion, log aggregation, event grouping and log file size 144 reduction techniques. 146 Feedback and discussion are welcome at https://github.com/quicwg/qlog 147 (https://github.com/quicwg/qlog). Readers are advised to refer to 148 the "editor's draft" at that URL for an up-to-date version of this 149 document. 151 Concrete examples of integrations of this schema in various 152 programming languages can be found at https://github.com/quiclog/ 153 qlog/ (https://github.com/quiclog/qlog/). 155 1.1. Notational Conventions 157 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 158 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 159 document are to be interpreted as described in [RFC2119]. 161 While the qlog schema's are format-agnostic, for readability the qlog 162 documents will use a JSON-inspired format ([RFC8259]) for examples 163 and definitions. 165 As qlog can be serialized both textually but also in binary, we 166 employ a custom datatype definition language, inspired loosely by the 167 "TypeScript" language (https://www.typescriptlang.org/). 169 This document describes how to employ JSON and NDJSON as textual 170 serializations for qlog in Section 6. Other documents will describe 171 how to utilize other concrete serialization options, though tips and 172 requirements for these are also listed in this document (Section 6). 174 The main general conventions in this document a reader should be 175 aware of are: 177 * obj? : this object is optional 179 * type1 | type2 : a union of these two types (object can be either 180 type1 OR type2) 182 * obj:type : this object has this concrete type 184 * obj:array : this object is an array of this type 186 * class : defines a new type 188 * // : single-line comment 190 The main data types are: 192 * int8 : signed 8-bit integer 193 * int16 : signed 16-bit integer 195 * int32 : signed 32-bit integer 197 * int64 : signed 64-bit integer 199 * uint8 : unsigned 8-bit integer 201 * uint16 : unsigned 16-bit integer 203 * uint32 : unsigned 32-bit integer 205 * uint64 : unsigned 64-bit integer 207 * float : 32-bit floating point value 209 * double : 64-bit floating point value 211 * byte : an individual raw byte (8-bit) value (use array or 212 the shorthand "bytes" to specify a binary blob) 214 * string : list of Unicode (typically UTF-8) encoded characters 216 * boolean : boolean 218 * enum: fixed list of values (Unless explicity defined, the value of 219 an enum entry is the string version of its name (e.g., initial = 220 "initial")) 222 * any : represents any object type. Mainly used here as a 223 placeholder for more concrete types defined in related documents 224 (e.g., specific event types) 226 All timestamps and time-related values (e.g., offsets) in qlog are 227 logged as doubles in the millisecond resolution. 229 Other qlog documents can define their own data types (e.g., 230 separately for each Packet type that a protocol supports). 232 2. Design goals 234 The main tenets for the qlog schema design are: 236 * Streamable, event-based logging 238 * Flexibility in the format, complexity in the tooling (e.g., few 239 components are a MUST, tools need to deal with this) 241 * Extensible and pragmatic (e.g., no complex fixed schema with 242 extension points) 244 * Aggregation and transformation friendly (e.g., the top-level 245 element is a container for individual traces, group_id can be used 246 to tag events to a particular context) 248 * Metadata is stored together with event data 250 3. The high level qlog schema 252 A qlog file should be able to contain several indivdual traces and 253 logs from multiple vantage points that are in some way related. To 254 that end, the top-level element in the qlog schema defines only a 255 small set of "header" fields and an array of component traces. For 256 this document, the required "qlog_version" field MUST have a value of 257 "qlog-03-WIP". 259 Note: there have been several previously broadly deployed qlog 260 versions based on older drafts of this document (see draft-marx- 261 qlog-main-schema). The old values for the "qlog_version" field 262 were "draft-00", "draft-01" and "draft-02". When qlog was moved 263 to the QUIC working group, we decided to increment the existing 264 counter, rather than reverting back to -00. As such, any 265 numbering indicating in the "qlog_version" field is explicitly not 266 tied to a particular version of the draft documents. 268 As qlog can be serialized in a variety of ways, the "qlog_format" 269 field is used to indicate which serialization option was chosen. Its 270 value MUST either be one of the options defined in this document 271 (e.g., Section 6) or the field must be omitted entirely, in which 272 case it assumes the default value of "JSON". 274 In order to make it easier to parse and identify qlog files and their 275 serialization format, the "qlog_version" and "qlog_format" fields and 276 their values SHOULD be in the first 256 characters/bytes of the 277 resulting log file. 279 An example of the qlog file's top-level structure is shown in 280 Figure 1. 282 Definition: 284 class QlogFile { 285 qlog_version:string, 286 qlog_format?:string, 287 title?:string, 288 description?:string, 289 summary?: Summary, 290 traces: array 291 } 293 JSON serialization: 295 { 296 "qlog_version": "draft-03-WIP", 297 "qlog_format": "JSON", 298 "title": "Name of this particular qlog file (short)", 299 "description": "Description for this group of traces (long)", 300 "summary": { 301 ... 302 }, 303 "traces": [...] 304 } 306 Figure 1: Top-level element 308 3.1. summary 310 In a real-life deployment with a large amount of generated logs, it 311 can be useful to sort and filter logs based on some basic summarized 312 or aggregated data (e.g., log length, packet loss rate, log location, 313 presence of error events, ...). The summary field (if present) 314 SHOULD be on top of the qlog file, as this allows for the file to be 315 processed in a streaming fashion (i.e., the implementation could just 316 read up to and including the summary field and then only load the 317 full logs that are deemed interesting by the user). 319 As the summary field is highly deployment-specific, this document 320 does not specify any default fields or their semantics. Some 321 examples of potential entries are shown in Figure 2. 323 Definition (purely illustrative example): 325 class Summary { 326 "trace_count":uint32, // amount of traces in this file 327 "max_duration":uint64, // time duration of the longest trace in ms 328 "max_outgoing_loss_rate":float, // highest loss rate for outgoing packets over all traces 329 "total_event_count":uint64, // total number of events across all traces, 330 "error_count":uint64 // total number of error events in this trace 331 } 333 JSON serialization: 335 { 336 "trace_count": 1, 337 "max_duration": 5006, 338 "max_outgoing_loss_rate": 0.013, 339 "total_event_count": 568, 340 "error_count": 2 341 } 343 Figure 2: Summary example definition 345 3.2. traces 347 It is often advantageous to group several related qlog traces 348 together in a single file. For example, we can simultaneously 349 perform logging on the client, on the server and on a single point on 350 their common network path. For analysis, it is useful to aggregate 351 these three individual traces together into a single file, so it can 352 be uniquely stored, transferred and annotated. 354 As such, the "traces" array contains a list of individual qlog 355 traces. Typical qlogs will only contain a single trace in this 356 array. These can later be combined into a single qlog file by taking 357 the "traces" entry/entries for each qlog file individually and 358 copying them to the "traces" array of a new, aggregated qlog file. 359 This is typically done in a post-processing step. 361 The "traces" array can thus contain both normal traces (for the 362 definition of the Trace type, see Section 3.3), but also "error" 363 entries. These indicate that we tried to find/convert a file for 364 inclusion in the aggregated qlog, but there was an error during the 365 process. Rather than silently dropping the erroneous file, we can 366 opt to explicitly include it in the qlog file as an entry in the 367 "traces" array, as shown in Figure 3. 369 Definition: 371 class TraceError { 372 error_description: string, // A description of the error 373 uri?: string, // the original URI at which we attempted to find the file 374 vantage_point?: VantagePoint // see {{vantage_point}}: the vantage point we were expecting to include here 375 } 377 JSON serialization: 379 { 380 "error_description": "File could not be found", 381 "uri": "/srv/traces/today/latest.qlog", 382 "vantage_point": { type: "server" } 383 } 385 Figure 3: TraceError definition 387 Note that another way to combine events of different traces in a 388 single qlog file is through the use of the "group_id" field, 389 discussed in Section 3.4.6. 391 3.3. Individual Trace containers 393 The exact conceptual definition of a Trace can be fluid. For 394 example, a trace could contain all events for a single connection, 395 for a single endpoint, for a single measurement interval, for a 396 single protocol, etc. As such, a Trace container contains some 397 metadata in addition to the logged events, see Figure 4. 399 In the normal use case however, a trace is a log of a single data 400 flow collected at a single location or vantage point. For example, 401 for QUIC, a single trace only contains events for a single logical 402 QUIC connection for either the client or the server. 404 The semantics and context of the trace can mainly be deduced from the 405 entries in the "common_fields" list and "vantage_point" field. 407 Definition: 409 class Trace { 410 title?: string, 411 description?: string, 412 configuration?: Configuration, 413 common_fields?: CommonFields, 414 vantage_point: VantagePoint, 415 events: array 416 } 418 JSON serialization: 420 { 421 "title": "Name of this particular trace (short)", 422 "description": "Description for this trace (long)", 423 "configuration": { 424 "time_offset": 150 425 }, 426 "common_fields": { 427 "ODCID": "abcde1234", 428 "time_format": "absolute" 429 }, 430 "vantage_point": { 431 "name": "backend-67", 432 "type": "server" 433 }, 434 "events": [...] 435 } 437 Figure 4: Trace container definition 439 3.3.1. configuration 441 We take into account that a qlog file is usually not used in 442 isolation, but by means of various tools. Especially when 443 aggregating various traces together or preparing traces for a 444 demonstration, one might wish to persist certain tool-based settings 445 inside the qlog file itself. For this, the configuration field is 446 used. 448 The configuration field can be viewed as a generic metadata field 449 that tools can fill with their own fields, based on per-tool logic. 450 It is best practice for tools to prefix each added field with their 451 tool name to prevent collisions across tools. This document only 452 defines two optional, standard, tool-independent configuration 453 settings: "time_offset" and "original_uris". 455 Definition: 457 class Configuration { 458 time_offset:double, // in ms, 459 original_uris: array, 461 // list of fields with any type 462 } 464 JSON serialization: 466 { 467 "time_offset": 150, // starts 150ms after the first timestamp indicates 468 "original_uris": [ 469 "https://example.org/trace1.qlog", 470 "https://example.org/trace2.qlog" 471 ] 472 } 474 Figure 5: Configuration definition 476 3.3.1.1. time_offset 478 The time_offset field indicates by how many milliseconds the starting 479 time of the current trace should be offset. This is useful when 480 comparing logs taken from various systems, where clocks might not be 481 perfectly synchronous. Users could use manual tools or automated 482 logic to align traces in time and the found optimal offsets can be 483 stored in this field for future usage. The default value is 0. 485 3.3.1.2. original_uris 487 The original_uris field is used when merging multiple individual qlog 488 files or other source files (e.g., when converting .pcaps to qlog). 489 It allows to keep better track where certain data came from. It is a 490 simple array of strings. It is an array instead of a single string, 491 since a single qlog trace can be made up out of an aggregation of 492 multiple component qlog traces as well. The default value is an 493 empty array. 495 3.3.1.3. custom fields 497 Tools can add optional custom metadata to the "configuration" field 498 to store state and make it easier to share specific data viewpoints 499 and view configurations. 501 Two examples from the qvis toolset (https://qvis.edm.uhasselt.be) are 502 shown in Figure 6. 504 { 505 "configuration" : { 506 "qvis" : { 507 // when loaded into the qvis toolsuite's congestion graph tool 508 // zoom in on the period between 1s and 2s and select the 124th event defined in this trace 509 "congestion_graph": { 510 "startX": 1000, 511 "endX": 2000, 512 "focusOnEventIndex": 124 513 } 515 // when loaded into the qvis toolsuite's sequence diagram tool 516 // automatically scroll down the timeline to the 555th event defined in this trace 517 "sequence_diagram" : { 518 "focusOnEventIndex": 555 519 } 520 } 521 } 522 } 524 Figure 6: Custom configuration fields example 526 3.3.2. vantage_point 528 The vantage_point field describes the vantage point from which the 529 trace originates, see Figure 7. Each trace can have only a single 530 vantage_point and thus all events in a trace MUST BE from the 531 perspective of this vantage_point. To include events from multiple 532 vantage_points, implementers can for example include multiple traces, 533 split by vantage_point, in a single qlog file. 535 Definition: 537 class VantagePoint { 538 name?: string, 539 type: VantagePointType, 540 flow?: VantagePointType 541 } 543 class VantagePointType { 544 server, // endpoint which initiates the connection. 545 client, // endpoint which accepts the connection. 546 network, // observer in between client and server. 547 unknown 548 } 550 JSON serialization examples: 552 { 553 "name": "aioquic client", 554 "type": "client", 555 } 557 { 558 "name": "wireshark trace", 559 "type": "network", 560 "flow": "client" 561 } 563 Figure 7: VantagePoint definition 565 The flow field is only required if the type is "network" (for 566 example, the trace is generated from a packet capture). It is used 567 to disambiguate events like "packet sent" and "packet received". 568 This is indicated explicitly because for multiple reasons (e.g., 569 privacy) data from which the flow direction can be otherwise inferred 570 (e.g., IP addresses) might not be present in the logs. 572 Meaning of the different values for the flow field: * "client" 573 indicates that this vantage point follows client data flow semantics 574 (a "packet sent" event goes in the direction of the server). * 575 "server" indicates that this vantage point follow server data flow 576 semantics (a "packet sent" event goes in the direction of the 577 client). * "unknown" indicates that the flow's direction is unknown. 579 Depending on the context, tools confronted with "unknown" values in 580 the vantage_point can either try to heuristically infer the semantics 581 from protocol-level domain knowledge (e.g., in QUIC, the client 582 always sends the first packet) or give the user the option to switch 583 between client and server perspectives manually. 585 3.4. Field name semantics 587 Inside of the "events" field of a qlog trace is a list of events 588 logged by the endpoint. Each event is specified as a generic object 589 with a number of member fields and their associated data. Depending 590 on the protocol and use case, the exact member field names and their 591 formats can differ across implementations. This section lists the 592 main, pre-defined and reserved field names with specific semantics 593 and expected corresponding value formats. 595 Each qlog event at minimum requires the "time" (Section 3.4.1), 596 "name" (Section 3.4.2) and "data" (Section 3.4.3) fields. Other 597 typical fields are "time_format" (Section 3.4.1), "protocol_type" 598 (Section 3.4.4), "trigger" (Section 3.4.5), and "group_id" 599 Section 3.4.6. As especially these later fields typically have 600 identical values across individual event instances, they are normally 601 logged separately in the "common_fields" (Section 3.4.7). 603 The specific values for each of these fields and their semantics are 604 defined in separate documents, specific per protocol or use case. 605 For example: event definitions for QUIC, HTTP/3 and QPACK can be 606 found in [QLOG-QUIC] and [QLOG-H3]. 608 Other fields are explicitly allowed by the qlog approach, and tools 609 SHOULD allow for the presence of unknown event fields, but their 610 semantics depend on the context of the log usage (e.g., for QUIC, the 611 ODCID field is used), see [QLOG-QUIC]. 613 An example of a qlog event with its component fields is shown in 614 Figure 8. 616 Definition: 618 class Event { 619 time: double, 620 name: string, 621 data: any, 623 protocol_type?: Array, 624 group_id?: string|uint32, 626 time_format?: "absolute"|"delta"|"relative", 628 // list of fields with any type 629 } 631 JSON serialization: 633 { 634 time: 1553986553572, 636 name: "transport:packet_sent", 637 data: { ... } 639 protocol_type: ["QUIC","HTTP3"], 640 group_id: "127ecc830d98f9d54a42c4f0842aa87e181a", 642 time_format: "absolute", 644 ODCID: "127ecc830d98f9d54a42c4f0842aa87e181a", // QUIC specific 645 } 647 Figure 8: Event fields definition 649 3.4.1. timestamps 651 The "time" field indicates the timestamp at which the event occured. 652 Its value is typically the Unix timestamp since the 1970 epoch 653 (number of milliseconds since midnight UTC, January 1, 1970, ignoring 654 leap seconds). However, qlog supports two more succint timestamps 655 formats to allow reducing file size. The employed format is 656 indicated in the "time_format" field, which allows one of three 657 values: "absolute", "delta" or "relative": 659 * Absolute: Include the full absolute timestamp with each event. 660 This approach uses the largest amount of characters. This is also 661 the default value of the "time_format" field. 663 * Delta: Delta-encode each time value on the previously logged 664 value. The first event in a trace typically logs the full 665 absolute timestamp. This approach uses the least amount of 666 characters. 668 * Relative: Specify a full "reference_time" timestamp (typically 669 this is done up-front in "common_fields", see Section 3.4.7) and 670 include only relatively-encoded values based on this 671 reference_time with each event. The "reference_time" value is 672 typically the first absolute timestamp. This approach uses a 673 medium amount of characters. 675 The first option is good for stateless loggers, the second and third 676 for stateful loggers. The third option is generally preferred, since 677 it produces smaller files while being easier to reason about. An 678 example for each option can be seen in Figure 9. 680 The absolute approach will use: 681 1500, 1505, 1522, 1588 683 The delta approach will use: 684 1500, 5, 17, 66 686 The relative approach will: 687 - set the reference_time to 1500 in "common_fields" 688 - use: 0, 5, 22, 88 690 Figure 9: Three different approaches for logging timestamps 692 One of these options is typically chosen for the entire trace (put 693 differently: each event has the same value for the "time_format" 694 field). Each event MUST include a timestamp in the "time" field. 696 Events in each individual trace SHOULD be logged in strictly 697 ascending timestamp order (though not necessarily absolute value, for 698 the "delta" format). Tools CAN sort all events on the timestamp 699 before processing them, though are not required to (as this could 700 impose a significant processing overhead). This can be a problem 701 especially for multi-threaded and/or streaming loggers, who could 702 consider using a separate postprocesser to order qlog events in time 703 if a tool do not provide this feature. 705 Timestamps do not have to use the UNIX epoch timestamp as their 706 reference. For example for privacy considerations, any initial 707 reference timestamps (for example "endpoint uptime in ms" or "time 708 since connection start in ms") can be chosen. Tools SHOULD NOT 709 assume the ability to derive the absolute Unix timestamp from qlog 710 traces, nor allow on them to relatively order events across two or 711 more separate traces (in this case, clock drift should also be taken 712 into account). 714 3.4.2. category and event 716 Events differ mainly in the type of metadata associated with them. 717 To help identify a given event and how to interpret its metadata in 718 the "data" field (see Section 3.4.3), each event has an associated 719 "name" field. This can be considered as a concatenation of two other 720 fields, namely event "category" and event "type". 722 Category allows a higher-level grouping of events per specific event 723 type. For example for QUIC and HTTP/3, the different categories 724 could be "transport", "http", "qpack", and "recovery". Within these 725 categories, the event Type provides additional granularity. For 726 example for QUIC and HTTP/3, within the "transport" Category, there 727 would be "packet_sent" and "packet_received" events. 729 Logging category and type separately conceptually allows for fast and 730 high-level filtering based on category and the re-use of event types 731 across categories. However, it also considerably inflates the log 732 size and this flexibility is not used extensively in practice at the 733 time of writing. 735 As such, the default approach in qlog is to concatenate both field 736 values using the ":" character in the "name" field, as can be seen in 737 Figure 10. As such, qlog category and type names MUST NOT include 738 this character. 740 JSON serialization using separate fields: 741 { 742 category: "transport", 743 type: "packet_sent" 744 } 746 JSON serialization using ":" concatenated field: 747 { 748 name: "transport:packet_sent" 749 } 751 Figure 10: Ways of logging category, type and name of an event. 753 Certain serializations CAN emit category and type as separate fields, 754 and qlog tools SHOULD be able to deal with both the concatenated 755 "name" field, and the separate "category" and "type" fields. Text- 756 based serializations however are encouraged to employ the 757 concatenated "name" field for efficiency. 759 3.4.3. data 761 The data field is a generic object. It contains the per-event 762 metadata and its form and semantics are defined per specific sort of 763 event. For example, data field value definitons for QUIC and HTTP/3 764 can be found in [QLOG-QUIC] and [QLOG-H3]. 766 One purely illustrative example for a QUIC "packet_sent" event is 767 shown in Figure 11. 769 Definition: 771 class TransportPacketSentEvent { 772 packet_size?:uint32, 773 header:PacketHeader, 774 frames?:Array 775 } 777 JSON serialization: 779 { 780 packet_size: 1280, 781 header: { 782 packet_type: "1RTT", 783 packet_number: 123 784 }, 785 frames: [ 786 { 787 frame_type: "stream", 788 length: 1000, 789 offset: 456 790 }, 791 { 792 frame_type: "padding" 793 } 794 ] 795 } 797 Figure 11: Example of the 'data' field for a QUIC packet_sent event 799 3.4.4. protocol_type 801 The "protocol_type" array field indicates to which protocols (or 802 protocol "stacks") this event belongs. This allows a single qlog 803 file to aggregate traces of different protocols (e.g., a web server 804 offering both TCP+HTTP/2 and QUIC+HTTP/3 connections). 806 For example, QUIC and HTTP/3 events have the "QUIC" and "HTTP3" 807 protocol_type entry values, see [QLOG-QUIC] and [QLOG-H3]. 809 Typically however, all events in a single trace are of the same few 810 protocols, and this array field is logged once in "common_fields", 811 see Section 3.4.7. 813 3.4.5. triggers 815 Sometimes, additional information is needed in the case where a 816 single event can be caused by a variety of other events. In the 817 normal case, the context of the surrounding log messages gives a hint 818 as to which of these other events was the cause. However, in highly- 819 parallel and optimized implementations, corresponding log messages 820 might separated in time. Another option is to explicitly indicate 821 these "triggers" in a high-level way per-event to get more fine- 822 grained information without much additional overhead. 824 In qlog, the optional "trigger" field contains a string value 825 describing the reason (if any) for this event instance occuring. 826 While this "trigger" field could be a property of the qlog Event 827 itself, it is instead a property of the "data" field instead. This 828 choice was made because many event types do not include a trigger 829 value, and having the field at the Event-level would cause overhead 830 in some serializations. Additional information on the trigger can be 831 added in the form of additional member fields of the "data" field 832 value, yet this is highly implementation-specific, as are the trigger 833 field's string values. 835 One purely illustrative example of some potential triggers for QUIC's 836 "packet_dropped" event is shown in Figure 12. 838 Definition: 840 class QuicPacketDroppedEvent { 841 packet_type?:PacketType, 842 raw_length?:uint32, 844 trigger?: "key_unavailable" | "unknown_connection_id" | "decrypt_error" | "unsupported_version" 845 } 846 Figure 12: Trigger example 848 3.4.6. group_id 850 As discussed in Section 3.3, a single qlog file can contain several 851 traces taken from different vantage points. However, a single trace 852 from one endpoint can also contain events from a variety of sources. 853 For example, a server implementation might choose to log events for 854 all incoming connections in a single large (streamed) qlog file. As 855 such, we need a method for splitting up events belonging to separate 856 logical entities. 858 The simplest way to perform this splitting is by associating a "group 859 identifier" to each event that indicates to which conceptual "group" 860 each event belongs. A post-processing step can then extract events 861 per group. However, this group identifier can be highly protocol and 862 context-specific. In the example above, we might use QUIC's 863 "Original Destination Connection ID" to uniquely identify a 864 connection. As such, they might add a "ODCID" field to each event. 865 However, a middlebox logging IP or TCP traffic might rather use four- 866 tuples to identify connections, and add a "four_tuple" field. 868 As such, to provide consistency and ease of tooling in cross-protocol 869 and cross-context setups, qlog instead defines the common "group_id" 870 field, which contains a string value. Implementations are free to 871 use their preferred string serialization for this field, so long as 872 it contains a unique value per logical group. Some examples can be 873 seen in Figure 13. 875 JSON serialization for events grouped by four tuples and QUIC connection IDs: 877 events: [ 878 { 879 time: 1553986553579, 880 protocol_type: ["TCP", "TLS", "HTTP2"], 881 group_id: "ip1=2001:67c:1232:144:9498:6df6:f450:110b,ip2=2001:67c:2b0:1c1::198,port1=59105,port2=80", 882 name: "transport:packet_received", 883 data: { ... }, 884 }, 885 { 886 time: 1553986553581, 887 protocol_type: ["QUIC","HTTP3"], 888 group_id: "127ecc830d98f9d54a42c4f0842aa87e181a", 889 name: "transport:packet_sent", 890 data: { ... }, 891 } 892 ] 893 Figure 13: Example of group_id usage 895 Note that in some contexts (for example a Multipath transport 896 protocol) it might make sense to add additional contextual per-event 897 fields (for example "path_id"), rather than use the group_id field 898 for that purpose. 900 Note also that, typically, a single trace only contains events 901 belonging to a single logical group (for example, an individual QUIC 902 connection). As such, instead of logging the "group_id" field with 903 an identical value for each event instance, this field is typically 904 logged once in "common_fields", see Section 3.4.7. 906 3.4.7. common_fields 908 As discussed in the previous sections, information for a typical qlog 909 event varies in three main fields: "time", "name" and associated 910 data. Additionally, there are also several more advanced fields that 911 allow mixing events from different protocols and contexts inside of 912 the same trace (for example "protocol_type" and "group_id"). In most 913 "normal" use cases however, the values of these advanced fields are 914 consistent for each event instance (for example, a single trace 915 contains events for a single QUIC connection). 917 To reduce file size and making logging easier, qlog uses the 918 "common_fields" list to indicate those fields and their values that 919 are shared by all events in this component trace. This prevents 920 these fields from being logged for each individual event. An example 921 of this is shown in Figure 14. 923 JSON serialization with repeated field values per-event instance: 925 { 926 events: [{ 927 group_id: "127ecc830d98f9d54a42c4f0842aa87e181a", 928 protocol_type: ["QUIC","HTTP3"], 929 time_format: "relative", 930 reference_time: "1553986553572", 932 time: 2, 933 name: "transport:packet_received", 934 data: { ... } 935 },{ 936 group_id: "127ecc830d98f9d54a42c4f0842aa87e181a", 937 protocol_type: ["QUIC","HTTP3"], 938 time_format: "relative", 939 reference_time: "1553986553572", 941 time: 7, 942 name: "http:frame_parsed", 943 data: { ... } 944 } 945 ] 946 } 948 JSON serialization with repeated field values extracted to common_fields: 950 { 951 common_fields: { 952 group_id: "127ecc830d98f9d54a42c4f0842aa87e181a", 953 protocol_type: ["QUIC","HTTP3"], 954 time_format: "relative", 955 reference_time: "1553986553572" 956 }, 957 events: [ 958 { 959 time: 2, 960 name: "transport:packet_received", 961 data: { ... } 962 },{ 963 7, 964 name: "http:frame_parsed", 965 data: { ... } 966 } 967 ] 968 } 970 Figure 14: Example of common_fields usage 972 The "common_fields" field is a generic dictionary of key-value pairs, 973 where the key is always a string and the value can be of any type, 974 but is typically also a string or number. As such, unknown entries 975 in this dictionary MUST be disregarded by the user and tools (i.e., 976 the presence of an uknown field is explicitly NOT an error). 978 The list of default qlog fields that are typically logged in 979 common_fields (as opposed to as individual fields per event instance) 980 are: 982 * time_format 984 * reference_time 986 * protocol_type 988 * group_id 990 Tools MUST be able to deal with these fields being defined either on 991 each event individually or combined in common_fields. Note that if 992 at least one event in a trace has a different value for a given 993 field, this field MUST NOT be added to common_fields but instead 994 defined on each event individually. Good example of such fields are 995 "time" and "data", who are divergent by nature. 997 4. Guidelines for event definition documents 999 This document only defines the main schema for the qlog format. This 1000 is intended to be used together with specific, per-protocol event 1001 definitions that specify the name (category + type) and data needed 1002 for each individual event. This is with the intent to allow the qlog 1003 main schema to be easily re-used for several protocols. Examples 1004 include the QUIC event definitions [QLOG-QUIC] and HTTP/3 and QPACK 1005 event definitions [QLOG-H3]. 1007 This section defines some basic annotations and concepts the creators 1008 of event definition documents SHOULD follow to ensure a measure of 1009 consistency, making it easier for qlog implementers to extrapolate 1010 from one protocol to another. 1012 4.1. Event design guidelines 1014 TODO: pending QUIC working group discussion. This text reflects the 1015 initial (qlog draft 01 and 02) setup. 1017 There are several ways of defining qlog events. In practice, we have 1018 seen two main types used so far: a) those that map directly to 1019 concepts seen in the protocols (e.g., "packet_sent") and b) those 1020 that act as aggregating events that combine data from several 1021 possible protocol behaviours or code paths into one (e.g., 1022 "parameters_set"). The latter are typically used as a means to 1023 reduce the amount of unique event definitions, as reflecting each 1024 possible protocol event as a separate qlog entity would cause an 1025 explosion of event types. 1027 Additionally, logging duplicate data is typically prevented as much 1028 as possible. For example, packet header values that remain 1029 consistent across many packets are split into separate events (for 1030 example "spin_bit_updated" or "connection_id_updated" for QUIC). 1032 Finally, we have typically refrained from adding additional state 1033 change events if those state changes can be directly inferred from 1034 data on the wire (for example flow control limit changes) if the 1035 implementation is bug-free and spec-compliant. Exceptions have been 1036 made for common events that benefit from being easily identifiable or 1037 individually logged (for example "packets_acked"). 1039 4.2. Event importance indicators 1041 Depending on how events are designed, it may be that several events 1042 allow the logging of similar or overlapping data. For example the 1043 separate QUIC "connection_started" event overlaps with the more 1044 generic "connection_state_updated". In these cases, it is not always 1045 clear which event should be logged or used, and which event should 1046 take precedence if e.g., both are present and provide conflicting 1047 information. 1049 To aid in this decision making, we recommend that each event SHOULD 1050 have an "importance indicator" with one of three values, in 1051 decreasing order of importance and exptected usage: 1053 * Core 1055 * Base 1057 * Extra 1059 The "Core" events are the events that SHOULD be present in all qlog 1060 files for a given protocol. These are typically tied to basic packet 1061 and frame parsing and creation, as well as listing basic internal 1062 metrics. Tool implementers SHOULD expect and add support for these 1063 events, though SHOULD NOT expect all Core events to be present in 1064 each qlog trace. 1066 The "Base" events add additional debugging options and CAN be present 1067 in qlog files. Most of these can be implicitly inferred from data in 1068 Core events (if those contain all their properties), but for many it 1069 is better to log the events explicitly as well, making it clearer how 1070 the implementation behaves. These events are for example tied to 1071 passing data around in buffers, to how internal state machines change 1072 and help show when decisions are actually made based on received 1073 data. Tool implementers SHOULD at least add support for showing the 1074 contents of these events, if they do not handle them explicitly. 1076 The "Extra" events are considered mostly useful for low-level 1077 debugging of the implementation, rather than the protocol. They 1078 allow more fine-grained tracking of internal behaviour. As such, 1079 they CAN be present in qlog files and tool implementers CAN add 1080 support for these, but they are not required to. 1082 Note that in some cases, implementers might not want to log for 1083 example data content details in the "Core" events due to performance 1084 or privacy considerations. In this case, they SHOULD use (a subset 1085 of) relevant "Base" events instead to ensure usability of the qlog 1086 output. As an example, implementations that do not log QUIC 1087 "packet_received" events and thus also not which (if any) ACK frames 1088 the packet contains, SHOULD log "packets_acked" events instead. 1090 Finally, for event types whose data (partially) overlap with other 1091 event types' definitions, where necessary the event definition 1092 document should include explicit guidance on which to use in specific 1093 situations. 1095 4.3. Custom fields 1097 Event definition documents are free to define new category and event 1098 types, top-level fields (e.g., a per-event field indicating its 1099 privacy properties or path_id in multipath protocols), as well as 1100 values for the "trigger" property within the "data" field, or other 1101 member fields of the "data" field, as they see fit. 1103 They however SHOULD NOT expect non-specialized tools to recognize or 1104 visualize this custom data. However, tools SHOULD make an effort to 1105 visualize even unknown data if possible in the specific tool's 1106 context. If they do not, they MUST ignore these unknown fields. 1108 5. Generic events and data classes 1110 There are some event types and data classes that are common across 1111 protocols, applications and use cases that benefit from being defined 1112 in a single location. This section specifies such common 1113 definitions. 1115 5.1. Raw packet and frame information 1117 While qlog is a more high-level logging format, it also allows the 1118 inclusion of most raw wire image information, such as byte lengths 1119 and even raw byte values. This can be useful when for example 1120 investigating or tuning packetization behaviour or determining 1121 encoding/framing overheads. However, these fields are not always 1122 necessary and can take up considerable space if logged for each 1123 packet or frame. They can also have a considerable privacy and 1124 security impact. As such, they are grouped in a separate optional 1125 field called "raw" of type RawInfo (where applicable). 1127 class RawInfo { 1128 length?:uint64; // the full byte length of the entity (e.g., packet or frame) including headers and trailers 1129 payload_length?:uint64; // the byte length of the entity's payload, without headers or trailers 1131 data?:bytes; // the contents of the full entity, including headers and trailers 1132 } 1134 Note: The RawInfo:data field can be truncated for privacy or 1135 security purposes (for example excluding payload data). In this 1136 case, the length properties should still indicate the non- 1137 truncated lengths. 1139 Note: We do not specify explicit header_length or trailer_length 1140 fields. In most protocols, header_length can be calculated by 1141 subtracing the payload_length from the length (e.g., if 1142 trailer_length is always 0). In protocols with trailers (e.g., 1143 QUIC's AEAD tag), event definitions documents SHOULD define other 1144 ways of logging the trailer_length to make the header_length 1145 calculation possible. 1147 The exact definitions entities, headers, trailers and payloads 1148 depend on the protocol used. If this is non-trivial, event 1149 definitions documents SHOULD include a clear explanation of how 1150 entities are mapped into the RawInfo structure. 1152 Note: Relatedly, many modern protocols use Variable-Length Integer 1153 Encoded (VLIE) values in their headers, which are of a dynamic 1154 length. Because of this, we cannot deterministally reconstruct 1155 the header encoding/length from non-RawInfo qlog data, as 1156 implementations might not necessarily employ the most efficient 1157 VLIE scheme for all values. As such, to make exact size-analysis 1158 possible, implementers should use explicit lengths in RawInfo 1159 rather than reconstructing them from other qlog data. Similarly, 1160 tool developers should only utilize RawInfo (and related 1161 information) in such tools to prevent errors. 1163 5.2. Generic events 1165 In typical logging setups, users utilize a discrete number of well- 1166 defined logging categories, levels or severities to log freeform 1167 (string) data. This generic events category replicates this approach 1168 to allow implementations to fully replace their existing text-based 1169 logging by qlog. This is done by providing events to log generic 1170 strings for the typical well-known logging levels (error, warning, 1171 info, debug, verbose). 1173 For the events defined below, the "category" is "generic" and their 1174 "type" is the name of the heading in lowercase (e.g., the "name" of 1175 the error event is "generic:error"). 1177 5.2.1. error 1179 Importance: Core 1181 Used to log details of an internal error that might not get reflected 1182 on the wire. 1184 Data: 1186 { 1187 code?:uint32, 1188 message?:string 1189 } 1191 5.2.2. warning 1193 Importance: Base 1195 Used to log details of an internal warning that might not get 1196 reflected on the wire. 1198 Data: 1200 { 1201 code?:uint32, 1202 message?:string 1203 } 1205 5.2.3. info 1207 Importance: Extra 1208 Used mainly for implementations that want to use qlog as their one 1209 and only logging format but still want to support unstructured string 1210 messages. 1212 Data: 1214 { 1215 message:string 1216 } 1218 5.2.4. debug 1220 Importance: Extra 1222 Used mainly for implementations that want to use qlog as their one 1223 and only logging format but still want to support unstructured string 1224 messages. 1226 Data: 1228 { 1229 message:string 1230 } 1232 5.2.5. verbose 1234 Importance: Extra 1236 Used mainly for implementations that want to use qlog as their one 1237 and only logging format but still want to support unstructured string 1238 messages. 1240 Data: 1242 { 1243 message:string 1244 } 1246 5.3. Simulation events 1248 When evaluating a protocol implementation, one typically sets up a 1249 series of interoperability or benchmarking tests, in which the test 1250 situations can change over time. For example, the network bandwidth 1251 or latency can vary during the test, or the network can be fully 1252 disable for a short time. In these setups, it is useful to know when 1253 exactly these conditions are triggered, to allow for proper 1254 correlation with other events. 1256 For the events defined below, the "category" is "simulation" and 1257 their "type" is the name of the heading in lowercase (e.g., the 1258 "name" of the scenario event is "simulation:scenario"). 1260 5.3.1. scenario 1262 Importance: Extra 1264 Used to specify which specific scenario is being tested at this 1265 particular instance. This could also be reflected in the top-level 1266 qlog's "summary" or "configuration" fields, but having a separate 1267 event allows easier aggregation of several simulations into one trace 1268 (e.g., split by "group_id"). 1270 { 1271 name?:string, 1272 details?:any 1273 } 1275 5.3.2. marker 1277 Importance: Extra 1279 Used to indicate when specific emulation conditions are triggered at 1280 set times (e.g., at 3 seconds in 2% packet loss is introduced, at 10s 1281 a NAT rebind is triggered). 1283 { 1284 type?:string, 1285 message?:string 1286 } 1288 6. Serializing qlog 1290 This document and other related qlog schema definitions are 1291 intentionally serialization-format agnostic. This means that 1292 implementers themselves can choose how to represent and serialize 1293 qlog data practically on disk or on the wire. Some examples of 1294 possible formats are JSON, CBOR, CSV, protocol buffers, flatbuffers, 1295 etc. 1297 All these formats make certain tradeoffs between flexibility and 1298 efficiency, with textual formats like JSON typically being more 1299 flexible but also less efficient than binary formats like protocol 1300 buffers. The format choice will depend on the practical use case of 1301 the qlog user. For example, for use in day to day debugging, a 1302 plaintext readable (yet relatively large) format like JSON is 1303 probably preferred. However, for use in production, a more optimized 1304 yet restricted format can be better. In this latter case, it will be 1305 more difficult to achieve interoperability between qlog 1306 implementations of various protocol stacks, as some custom or tweaked 1307 events from one might not be compatible with the format of the other. 1308 This will also reflect in tooling: not all tools will support all 1309 formats. 1311 This being said, the authors prefer JSON as the basis for storing 1312 qlog, as it retains full flexibility and maximum interoperability. 1313 Storage overhead can be managed well in practice by employing 1314 compression. For this reason, this document details both how to 1315 practically transform qlog schema definitions to JSON and to the 1316 streamable NDJSON. We discuss concrete options to bring down JSON 1317 size and processing overheads in Section 6.3. 1319 As depending on the employed format different deserializers/parsers 1320 should be used, the "qlog_format" field is used to indicate the 1321 chosen serialization approach. This field is always a string, but 1322 can be made hierarchical by the use of the "." separator between 1323 entries. For example, a value of "JSON.optimizationA" can indicate 1324 that a default JSON format is being used, but that a certain 1325 optimization of type A was applied to the file as well (see also 1326 Section 6.3). 1328 6.1. qlog to JSON mapping 1330 When mapping qlog to normal JSON, the "qlog_format" field MUST have 1331 the value "JSON". This is also the default qlog serialization and 1332 default value of this field. 1334 To facilitate this mapping, the qlog documents employ a format that 1335 is close to pure JSON for its examples and data definitions. Still, 1336 as JSON is not a typed format, there are some practical peculiarities 1337 to observe. 1339 6.1.1. numbers 1341 While JSON has built-in support for integers up to 64 bits in size, 1342 not all JSON parsers do. For example, none of the major Web browsers 1343 support full 64-bit integers at this time, as all numerical values 1344 (both floating-point numbers and integers) are internally represented 1345 as floating point IEEE 754 (https://en.wikipedia.org/wiki/Floating- 1346 point_arithmetic) values. In practice, this limits their integers to 1347 a maximum value of 2^53-1. Integers larger than that are either 1348 truncated or produce a JSON parsing error. While this is expected to 1349 improve in the future (as "BigInt" support 1350 (https://developer.mozilla.org/en- 1351 US/docs/Web/JavaScript/Reference/Global_Objects/BigInt) has been 1352 introduced in most Browsers, though not yet integrated into JSON 1353 parsers), we still need to deal with it here. 1355 When transforming an int64, uint64 or double from qlog to JSON, the 1356 implementer can thus choose to either log them as JSON numbers 1357 (taking the risk of truncation or un-parseability) or to log them as 1358 strings instead. Logging as strings should however only be 1359 practically needed if the value is likely to exceed 2^53-1. In 1360 practice, even though protocols such as QUIC allow 64-bit values for 1361 for example stream identifiers, these high numbers are unlikely to be 1362 reached for the overwhelming majority of cases. As such, it is 1363 probably a valid trade-off to take the risk and log 64-bit values as 1364 JSON numbers instead of strings. 1366 Tools processing JSON-based qlog SHOULD however be able to deal with 1367 64-bit fields being serialized as either strings or numbers. 1369 6.1.2. bytes 1371 Unlike most binary formats, JSON does not allow the logging of raw 1372 binary blobs directly. As such, when serializing a byte or 1373 array, a scheme needs to be chosen. 1375 To represent qlog bytes in JSON, they MUST be serialized to their 1376 lowercase hexadecimal equivalents (with 0 prefix for values lower 1377 than 10). All values are directly appended to each other, without 1378 delimiters. The full value is not prefixed with 0x (as is sometimes 1379 common). An example is given in Figure 15. 1381 For the five raw unsigned byte input values of: 5 20 40 171 255, the JSON serialization is: 1383 { 1384 raw: "051428abff" 1385 } 1387 Figure 15: Example for serializing bytes 1389 As such, the resulting string will always have an even amount of 1390 characters and the original byte-size can be retrieved by dividing 1391 the string length by 2. 1393 6.1.2.1. Truncated values 1395 In some cases, it can be interesting not to log a full raw blob but 1396 instead a truncated value (for example, only the first 100 bytes of 1397 an HTTP response body to be able to discern which file it actually 1398 contained). In these cases, the original byte-size length cannot be 1399 obtained from the serialized value directly. As such, all qlog 1400 schema definitions SHOULD include a separate, length-indicating field 1401 for all fields of type array they specify. This allows always 1402 retrieving the original length, but also allows the omission of any 1403 raw value bytes of the field completely (e.g., out of privacy or 1404 security considerations). 1406 To reduce overhead however and in the case the full raw value is 1407 logged, the extra length-indicating field can be left out. As such, 1408 tools MUST be able to deal with this situation and derive the length 1409 of the field from the raw value if no separate length-indicating 1410 field is present. All possible permutations are shown by example in 1411 Figure 16. 1413 // both the full raw value and its length are present (length is redundant) 1414 { 1415 "raw_length": 5, 1416 "raw": "051428abff" 1417 } 1419 // only the raw value is present, indicating it represents the fields full value 1420 // the byte length is obtained by calculating raw.length / 2 1421 { 1422 "raw": "051428abff" 1423 } 1425 // only the length field is present, meaning the value was omitted 1426 { 1427 "raw_length": 5, 1428 } 1430 // both fields are present and the lengths do not match: the value was truncated to the first three bytes. 1431 { 1432 "raw_length": 5, 1433 "raw": "051428" 1434 } 1436 Figure 16: Example for serializing truncated bytes 1438 6.1.3. Summarizing table 1440 By definition, JSON strings are serialized surrounded by quotes. 1441 Numbers without. 1443 +===========+=====================================+ 1444 | qlog type | JSON type | 1445 +===========+=====================================+ 1446 | int8 | number | 1447 +-----------+-------------------------------------+ 1448 | int16 | number | 1449 +-----------+-------------------------------------+ 1450 | int32 | number | 1451 +-----------+-------------------------------------+ 1452 | uint8 | number | 1453 +-----------+-------------------------------------+ 1454 | uint16 | number | 1455 +-----------+-------------------------------------+ 1456 | uint32 | number | 1457 +-----------+-------------------------------------+ 1458 | float | number | 1459 +-----------+-------------------------------------+ 1460 | int64 | number or string | 1461 +-----------+-------------------------------------+ 1462 | uint64 | number or string | 1463 +-----------+-------------------------------------+ 1464 | double | number or string | 1465 +-----------+-------------------------------------+ 1466 | bytes | string (lowercase hex value) | 1467 +-----------+-------------------------------------+ 1468 | string | string | 1469 +-----------+-------------------------------------+ 1470 | boolean | string ("true" or "false") | 1471 +-----------+-------------------------------------+ 1472 | enum | string (full value/name, not index) | 1473 +-----------+-------------------------------------+ 1474 | any | object ( {...} ) | 1475 +-----------+-------------------------------------+ 1476 | array | array ( [...] ) | 1477 +-----------+-------------------------------------+ 1479 Table 1 1481 6.1.4. Other JSON specifics 1483 JSON files by definition ([RFC8259]) MUST utilize the UTF-8 encoding, 1484 both for the file itself and the string values. 1486 Most JSON parsers strictly follow the JSON specification. This 1487 includes the rule that trailing comma's are not allowed. As it is 1488 frequently annoying to remove these trailing comma's when logging 1489 events in a streaming fashion, tool implementers SHOULD allow the 1490 last event entry of a qlog trace to be an empty object. This allows 1491 loggers to simply close the qlog file by appending "{}]}]}" after 1492 their last added event. 1494 Finally, while not specifically required by the JSON specification, 1495 all qlog field names in a JSON serialization MUST be lowercase. 1497 6.2. qlog to NDJSON mapping 1499 One of the downsides of using pure JSON is that it is inherently a 1500 non-streamable format. Put differently, it is not possible to simply 1501 append new qlog events to a log file without "closing" this file at 1502 the end by appending "]}]}". Without these closing tags, most JSON 1503 parsers will be unable to parse the file entirely. As most platforms 1504 do not provide a standard streaming JSON parser (which would be able 1505 to deal with this problem), this document also provides a qlog 1506 mapping to a streamable JSON format called Newline-Delimited JSON 1507 (NDJSON) (http://ndjson.org/). 1509 When mapping qlog to NDJSON, the "qlog_format" field MUST have the 1510 value "NDJSON". 1512 NDJSON is very similar to JSON, except that it interprets each line 1513 in a file as a fully separate JSON object. Put differently, unlike 1514 default JSON, it does not require a file to be wrapped as a full 1515 object with "{ ... }" or "[ ... ]". Using this setup, qlog events 1516 can simply be appended as individually serialized lines at the back 1517 of a streamed logging file. 1519 For this to work, some qlog definitions have to be adjusted however. 1520 Mainly, events are no longer part of the "events" array in the Trace 1521 object, but are instead logged separately from the qlog "file header" 1522 (QlogFile class in Section 3). Additionally, qlog's NDJSON mapping 1523 does not allow logging multiple individual traces in a single qlog 1524 file. As such, the QlogFile:traces field is replaced by the singular 1525 "trace" field, which simply contains the Trace data directly. An 1526 example can be seen in Figure 17. Note that the "group_id" field can 1527 still be used on a per-event basis to include events from 1528 conceptually different sources in a single NDJSON qlog file. 1530 Note as well from Figure 17 that the file's header (QlogFileNDJSON) 1531 also needs to be fully serialized on a single line to be NDJSON 1532 compatible. 1534 Definition: 1536 class QlogFileNDJSON { 1537 qlog_format: "NDJSON", 1539 qlog_version:string, 1540 title?:string, 1541 description?:string, 1542 summary?: Summary, 1543 trace: Trace 1544 } 1545 // list of qlog events, separated by newlines 1547 NDJSON serialization: 1549 {"qlog_format":"NDJSON","qlog_version":"draft-03-WIP","title":"Name of this particular NDJSON qlog file (short)","description":"Description for this NDJSON qlog file (long)","trace":{"common_fields":{"protocol_type": ["QUIC","HTTP3"],"group_id":"127ecc830d98f9d54a42c4f0842aa87e181a","time_format":"relative","reference_time":"1553986553572"},"vantage_point":{"name":"backend-67","type":"server"}}} 1550 {"time": 2, "name": "transport:packet_received", "data": { ... } } 1551 {"time": 7, "name": "http:frame_parsed", "data": { ... } } 1553 Figure 17: Top-level element 1555 Finally, while not specifically required by the NDJSON specification, 1556 all qlog field names in a NDJSON serialization MUST be lowercase. 1558 6.2.1. Supporting NDJSON in tooling 1560 Note that NDJSON is not supported in most default programming 1561 environments (unlike normal JSON). However, several custom NDJSON 1562 parsing libraries exist (http://ndjson.org/libraries.html) that can 1563 be used and the format is easy enough to parse with existing 1564 implementations (i.e., by splitting the file into its component lines 1565 and feeding them to a normal JSON parser individually, as each line 1566 by itself is a valid JSON object). 1568 6.3. Other optimizated formatting options 1570 Both the JSON and NDJSON formatting options described above are 1571 serviceable in general small to medium scale (debugging) setups. 1572 However, these approaches tend to be relatively verbose, leading to 1573 larger file sizes. Additionally, generalized (ND)JSON 1574 (de)serialization performance is typically (slightly) lower than that 1575 of more optimized and predictable formats. Both aspects make these 1576 formats more challenging (though still practical 1577 (https://qlog.edm.uhasselt.be/anrw/)) to use in large scale setups. 1579 During the development of qlog, we compared a multitude of 1580 alternative formatting and optimization options. The results of this 1581 study are summarized on the qlog github repository 1582 (https://github.com/quiclog/internet-drafts/issues/30#issuecomment- 1583 617675097). The rest of this section discusses some of these 1584 approaches implementations could choose and the expected gains and 1585 tradeoffs inherent therein. Tools SHOULD support mainly the 1586 compression options listed in Section 6.3.2, as they provide the 1587 largest wins for the least cost overall. 1589 Over time, specific qlog formats and encodings can be created that 1590 more formally define and combine some of the discussed optimizations 1591 or add new ones. We choose to define these schemes in separate 1592 documents to keep the main qlog definition clean and generalizable, 1593 as not all contexts require the same performance or flexibility as 1594 others and qlog is intended to be a broadly usable and extensible 1595 format (for example more flexibility is needed in earlier stages of 1596 protocol development, while more performance is typically needed in 1597 later stages). This is also the main reason why the general qlog 1598 format is the less optimized JSON instead of a more performant 1599 option. 1601 To be able to easily distinguish between these options in qlog 1602 compatible tooling (without the need to have the user provide out-of- 1603 band information or to (heuristically) parse and process files in a 1604 multitude of ways, see also Section 8), we recommend using explicit 1605 file extensions to indicate specific formats. As there are no 1606 standards in place for this type of extension to format mapping, we 1607 employ a commonly used scheme here. Our approach is to list the 1608 applied optimizations in the extension in ascending order of 1609 application (e.g., if a qlog file is first optimized with technique A 1610 and then compressed with technique B, the resulting file would have 1611 the extension ".qlog.A.B"). This allows tooling to start at the back 1612 of the extension to "undo" applied optimizations to finally arrive at 1613 the expected qlog representation. 1615 6.3.1. Data structure optimizations 1617 The first general category of optimizations is to alter the 1618 representation of data within an (ND)JSON qlog file to reduce file 1619 size. 1621 The first option is to employ a scheme similar to the CSV (comma 1622 separated value [rfc4180]) format, which utilizes the concept of 1623 column "headers" to prevent repeating field names for each datapoint 1624 instance. Concretely for JSON qlog, several field names are repeated 1625 with each event (i.e., time, name, data). These names could be 1626 extracted into a separate list, after which qlog events could be 1627 serialized as an array of values, as opposed to a full object. This 1628 approach was a key part of the original qlog format (prior to draft 1629 02) using the "event_fields" field. However, tests showed that this 1630 optimization only provided a mean file size reduction of 5% (100MB to 1631 95MB) while significantly increasing the implementation complexity, 1632 and this approach was abandoned in favor of the default JSON setup. 1633 Implementations using this format should not employ a separate file 1634 extension (as it still uses JSON), but rather employ a new value of 1635 "JSON.namedheaders" (or "NDJSON.namedheaders") for the "qlog_format" 1636 field (see Section 3). 1638 The second option is to replace field values and/or names with 1639 indices into a (dynamic) lookup table. This is a common compression 1640 technique and can provide significant file size reductions (up to 50% 1641 in our tests, 100MB to 50MB). However, this approach is even more 1642 difficult to implement efficiently and requires either including the 1643 (dynamic) table in the resulting file (an approach taken by for 1644 example Chromium's NetLog format 1645 (https://www.chromium.org/developers/design-documents/network-stack/ 1646 netlog)) or defining a (static) table up-front and sharing this 1647 between implementations. Implementations using this approach should 1648 not employ a separate file extension (as it still uses JSON), but 1649 rather employ a new value of "JSON.dictionary" (or 1650 "NDJSON.dictionary") for the "qlog_format" field (see Section 3). 1652 As both options either proved difficult to implement, reduced qlog 1653 file readability, and provided too little improvement compared to 1654 other more straightforward options (for example Section 6.3.2), these 1655 schemes are not inherently part of qlog. 1657 6.3.2. Compression 1659 The second general category of optimizations is to utilize a 1660 (generic) compression scheme for textual data. As qlog in the 1661 (ND)JSON format typically contains a large amount of repetition, off- 1662 the-shelf (text) compression techniques typically succeed very well 1663 in bringing down file sizes (regularly with up to two orders of 1664 magnitude in our tests, even for "fast" compression levels). As 1665 such, utilizing compression is recommended before attempting other 1666 optimization options, even though this might (somewhat) increase 1667 processing costs due to the additional compression step. 1669 The first option is to use GZIP compression ([RFC1952]). This 1670 generic compression scheme provides multiple compression levels 1671 (providing a trade-off between compression speed and size reduction). 1672 Utilized at level 6 (a medium setting thought to be applicable for 1673 streaming compression of a qlog stream in commodity devices), gzip 1674 compresses qlog JSON files to 7% of their initial size on average 1675 (100MB to 7MB). For this option, the file extension .qlog.gz SHOULD 1676 BE used. The "qlog_format" field should still reflect the original 1677 JSON formatting of the qlog data (e.g., "JSON" or "NDJSON"). 1679 The second option is to use Brotli compression ([RFC7932]). While 1680 similar to gzip, this more recent compression scheme provides a 1681 better efficiency. It also allows multiple compression levels. 1682 Utilized at level 4 (a medium setting thought to be applicable for 1683 streaming compression of a qlog stream in commodity devices), brotli 1684 compresses qlog JSON files to 7% of their initial size on average 1685 (100MB to 7MB). For this option, the file extension .qlog.br SHOULD 1686 BE used. The "qlog_format" field should still reflect the original 1687 JSON formatting of the qlog data (e.g., "JSON" or "NDJSON"). 1689 Other compression algorithms of course exist (for example xz, zstd, 1690 and lz4). We mainly recommend gzip and brotli because of their 1691 tweakable behaviour and wide support in web-based environments, which 1692 we envision as the main tooling ecosystem (see also Section 8). 1694 6.3.3. Binary formats 1696 The third general category of optimizations is to use a more 1697 optimized (often binary) format instead of the textual JSON format. 1698 This approach inherently produces smaller files and often has better 1699 (de)serialization performance. However, the resultant files are no 1700 longer human readable and some formats require hard tradeoffs between 1701 flexibility for performance. 1703 The first option is to use the CBOR (Concise Binary Object 1704 Representation [rfc7049]) format. For our purposes, CBOR can be 1705 viewed as a straighforward binary variant of JSON. As such, existing 1706 JSON qlog files can be trivially converted to and from CBOR (though 1707 slightly more work is needed for NDJSON qlogs). While CBOR thus does 1708 retain the full qlog flexibility, it only provides a 25% file size 1709 reduction (100MB to 75MB) compared to textual (ND)JSON. As CBOR 1710 support in programming environments is not as widespread as that of 1711 textual JSON and the format lacks human readability, CBOR was not 1712 chosen as the default qlog format. For this option, the file 1713 extension .qlog.cbor SHOULD BE used. The "qlog_format" field should 1714 still reflect the original JSON formatting of the qlog data (e.g., 1715 "JSON" or "NDJSON"). 1717 A second option is to use a more specialized binary format, such as 1718 Protocol Buffers (https://developers.google.com/protocol-buffers) 1719 (protobuf). This format is battle-tested, has support for optional 1720 fields and has libraries in most programming languages. Still, it is 1721 significantly less flexible than textual JSON or CBOR, as it relies 1722 on a separate, pre-defined schema (a .proto file). As such, it it 1723 not possible to (easily) log new event types in protobuf files 1724 without adjusting this schema as well, which has its own practical 1725 challenges. As qlog is intended to be a flexible, general purpose 1726 format, this type of format was not chosen as its basic 1727 serialization. The lower flexibility does lead to significantly 1728 reduced file sizes. Our straightforward mapping of the qlog main 1729 schema and QUIC/HTTP3 event types to protobuf created qlog files 24% 1730 as large as the raw JSON equivalents (100MB to 24MB). For this 1731 option, the file extension .qlog.protobuf SHOULD BE used. The 1732 "qlog_format" field should reflect the different internal format, for 1733 example: "qlog_format": "protobuf". 1735 Note that binary formats can (and should) also be used in conjunction 1736 with compression (see Section 6.3.2). For example, CBOR compresses 1737 well (to about 6% of the original textual JSON size (100MB to 6MB) 1738 for both gzip and brotli) and so does protobuf (5% (gzip) to 3% 1739 (brotli)). However, these gains are similar to the ones achieved by 1740 simply compression the textual JSON equivalents directly (7%, see 1741 Section 6.3.2). As such, since compression is still needed to 1742 achieve optimal file size reductions event with binary formats, we 1743 feel the more flexible compressed textual JSON options are a better 1744 default for the qlog format in general. 1746 6.3.4. Overview and summary 1748 In summary, textual JSON was chosen as the main qlog format due to 1749 its high flexibility and because its inefficiencies can be largely 1750 solved by the utilization of compression techniques (which are needed 1751 to achieve optimal results with other formats as well). 1753 Still, qlog implementers are free to define other qlog formats 1754 depending on their needs and context of use. These formats should be 1755 described in their own documents, the discussion in this document 1756 mainly acting as inspiration and high-level guidance. Implementers 1757 are encouraged to add concrete qlog formats and definitions to the 1758 designated public repository (https://github.com/quiclog/qlog). 1760 The following table provides an overview of all the discussed qlog 1761 formatting options with examples: 1763 +====================+=======================+================+ 1764 | format | qlog_format | extension | 1765 +====================+=======================+================+ 1766 | JSON Section 6.1 | JSON | .qlog | 1767 +--------------------+-----------------------+----------------+ 1768 | NDJSON Section 6.2 | NDJSON | .qlog | 1769 +--------------------+-----------------------+----------------+ 1770 | named headers | (ND)JSON.namedheaders | .qlog | 1771 | Section 6.3.1 | | | 1772 +--------------------+-----------------------+----------------+ 1773 | dictionary | (ND)JSON.dictionary | .qlog | 1774 | Section 6.3.1 | | | 1775 +--------------------+-----------------------+----------------+ 1776 | CBOR Section 6.3.3 | (ND)JSON | .qlog.cbor | 1777 +--------------------+-----------------------+----------------+ 1778 | protobuf | protobuf | .qlog.protobuf | 1779 | Section 6.3.3 | | | 1780 +--------------------+-----------------------+----------------+ 1781 +--------------------+-----------------------+----------------+ 1782 | gzip Section 6.3.2 | no change | .gz suffix | 1783 +--------------------+-----------------------+----------------+ 1784 | brotli | no change | .br suffix | 1785 | Section 6.3.2 | | | 1786 +--------------------+-----------------------+----------------+ 1788 Table 2 1790 6.4. Conversion between formats 1792 As discussed in the previous sections, a qlog file can be serialized 1793 in a multitude of formats, each of which can conceivably be 1794 transformed into or from one another without loss of information. 1795 For example, a number of NDJSON streamed qlogs could be combined into 1796 a JSON formatted qlog for later processing. Similarly, a captured 1797 binary qlog could be transformed to JSON for easier interpretation 1798 and sharing. 1800 Secondly, we can also consider other structured logging approaches 1801 that contain similar (though typically not identical) data to qlog, 1802 like raw packet capture files (for example .pcap files from tcpdump) 1803 or endpoint-specific logging formats (for example the NetLog format 1804 in Google Chrome). These are sometimes the only options, if an 1805 implementation cannot or will not support direct qlog output for any 1806 reason, but does provide other internal or external (e.g., 1807 SSLKEYLOGFILE export to allow decryption of packet captures) logging 1808 options For this second category, a (partial) transformation from/to 1809 qlog can also be defined. 1811 As such, when defining a new qlog serialization format or wanting to 1812 utilize qlog-compatible tools with existing codebases lacking qlog 1813 support, it is recommended to define and provide a concrete mapping 1814 from one format to default JSON-serialized qlog. Several of such 1815 mappings exist. Firstly, [pcap2qlog]((https://github.com/quiclog/ 1816 pcap2qlog) transforms QUIC and HTTP/3 packet capture files to qlog. 1817 Secondly, netlog2qlog 1818 (https://github.com/quiclog/qvis/tree/master/visualizations/src/ 1819 components/filemanager/netlogconverter) converts chromium's internal 1820 dictionary-encoded JSON format to qlog. Finally, quictrace2qlog 1821 (https://github.com/quiclog/quictrace2qlog) converts the older 1822 quictrace format to JSON qlog. Tools can then easily integrate with 1823 these converters (either by incorporating them directly or for 1824 example using them as a (web-based) API) so users can provide 1825 different file types with ease. For example, the qvis 1826 (https://qvis.edm.uhasselt.be) toolsuite supports a multitude of 1827 formats and qlog serializations. 1829 7. Methods of access and generation 1831 Different implementations will have different ways of generating and 1832 storing qlogs. However, there is still value in defining a few 1833 default ways in which to steer this generation and access of the 1834 results. 1836 7.1. Set file output destination via an environment variable 1838 To provide users control over where and how qlog files are created, 1839 we define two environment variables. The first, QLOGFILE, indicates 1840 a full path to where an individual qlog file should be stored. This 1841 path MUST include the full file extension. The second, QLOGDIR, sets 1842 a general directory path in which qlog files should be placed. This 1843 path MUST include the directory separator character at the end. 1845 In general, QLOGDIR should be preferred over QLOGFILE if an endpoint 1846 is prone to generate multiple qlog files. This can for example be 1847 the case for a QUIC server implementation that logs each QUIC 1848 connection in a separate qlog file. An alternative that uses 1849 QLOGFILE would be a QUIC server that logs all connections in a single 1850 file and uses the "group_id" field (Section 3.4.6) to allow post-hoc 1851 separation of events. 1853 Implementations SHOULD provide support for QLOGDIR and MAY provide 1854 support for QLOGFILE. 1856 When using QLOGDIR, it is up to the implementation to choose an 1857 appropriate naming scheme for the qlog files themselves. The chosen 1858 scheme will typically depend on the context or protocols used. For 1859 example, for QUIC, it is recommended to use the Original Destination 1860 Connection ID (ODCID), followed by the vantage point type of the 1861 logging endpoint. Examples of all options for QUIC are shown in 1862 Figure 18. 1864 Command: QLOGFILE=/srv/qlogs/client.qlog quicclientbinary 1866 Should result in the the quicclientbinary executable logging a single qlog file named client.qlog in the /srv/qlogs directory. 1867 This is for example useful in tests when the client sets up just a single connection and then exits. 1869 Command: QLOGDIR=/srv/qlogs/ quicserverbinary 1871 Should result in the quicserverbinary executable generating several logs files, one for each QUIC connection. 1872 Given two QUIC connections, with ODCID values "abcde" and "12345" respectively, this would result in two files: 1873 /srv/qlogs/abcde_server.qlog 1874 /srv/qlogs/12345_server.qlog 1876 Command: QLOGFILE=/srv/qlogs/server.qlog quicserverbinary 1878 Should result in the the quicserverbinary executable logging a single qlog file named server.qlog in the /srv/qlogs directory. 1879 Given that the server handled two QUIC connections before it was shut down, with ODCID values "abcde" and "12345" respectively, 1880 this would result in event instances in the qlog file being tagged with the "group_id" field with values "abcde" and "12345". 1882 Figure 18: Environment variable examples for a QUIC implementation 1884 7.2. Access logs via a well-known endpoint 1886 After generation, qlog implementers MAY make available generated logs 1887 and traces on an endpoint (typically the server) via the following 1888 .well-known URI: 1890 .well-known/qlog/IDENTIFIER.extension 1892 The IDENTIFIER variable depends on the context and the protocol. For 1893 example for QUIC, the lowercase Original Destination Connection ID 1894 (ODCID) is recommended, as it can uniquely identify a connection. 1895 Additionally, the extension depends on the chosen format (see 1896 Section 6.3.4). For example, for a QUIC connection with ODCID 1897 "abcde", the endpoint for fetching its default JSON-formatted .qlog 1898 file would be: 1900 .well-known/qlog/abcde.qlog 1902 Implementers SHOULD allow users to fetch logs for a given connection 1903 on a 2nd, separate connection. This helps prevent pollution of the 1904 logs by fetching them over the same connection that one wishes to 1905 observe through the log. Ideally, for the QUIC use case, the logs 1906 should also be approachable via an HTTP/2 or HTTP/1.1 endpoint (i.e., 1907 on TCP port 443), to for example aid debugging in the case where 1908 QUIC/UDP is blocked on the network. 1910 qlog implementers SHOULD NOT enable this .well-known endpoint in 1911 typical production settings to prevent (malicious) users from 1912 downloading logs from other connections. Implementers are advised to 1913 disable this endpoint by default and require specific actions from 1914 the end users to enable it (and potentially qlog itself). 1915 Implementers MUST also take into account the general privacy and 1916 security guidelines discussed in Section 9 before exposing qlogs to 1917 outside actors. 1919 8. Tooling requirements 1921 Tools ingestion qlog MUST indicate which qlog version(s), qlog 1922 format(s), compression methods and potentially other input file 1923 formats (for example .pcap) they support. Tools SHOULD at least 1924 support .qlog files in the default JSON format (Section 6.1). 1925 Additionally, they SHOULD indicate exactly which values for and 1926 properties of the name (category and type) and data fields they look 1927 for to execute their logic. Tools SHOULD perform a (high-level) 1928 check if an input qlog file adheres to the expected qlog schema. If 1929 a tool determines a qlog file does not contain enough supported 1930 information to correctly execute the tool's logic, it SHOULD generate 1931 a clear error message to this effect. 1933 Tools MUST NOT produce breaking errors for any field names and/or 1934 values in the qlog format that they do not recognize. Tools SHOULD 1935 indicate even unknown event occurences within their context (e.g., 1936 marking unknown events on a timeline for manual interpretation by the 1937 user). 1939 Tool authors should be aware that, depending on the logging 1940 implementation, some events will not always be present in all traces. 1941 For example, using a circular logging buffer of a fixed size, it 1942 could be that the earliest events (e.g., connection setup events) are 1943 later overwritten by "newer" events. Alternatively, some events can 1944 be intentionally omitted out of privacy or file size considerations. 1945 Tool authors are encouraged to make their tools robust enough to 1946 still provide adequate output for incomplete logs. 1948 9. Security and privacy considerations 1950 TODO : discuss privacy and security considerations (e.g., what NOT to 1951 log, what to strip out of a log before sharing, ...) 1953 TODO: strip out/don't log IPs, ports, specific CIDs, raw user data, 1954 exact times, HTTP HEADERS (or at least :path), SNI values 1956 TODO: see if there is merit in encrypting the logs and having the 1957 server choose an encryption key (e.g., sent in transport parameters) 1959 Good initial reference: Christian Huitema's blogpost 1960 (https://huitema.wordpress.com/2020/07/21/scrubbing-quic-logs-for- 1961 privacy/) 1963 10. IANA Considerations 1965 TODO: primarily the .well-known URI 1967 11. References 1969 11.1. Normative References 1971 [QLOG-H3] Marx, R., Ed., Niccolini, L., Ed., and M. Seemann, Ed., 1972 "HTTP/3 and QPACK event definitions for qlog", Work in 1973 Progress, Internet-Draft, draft-ietf-quic-qlog-h3-events- 1974 00, . 1977 [QLOG-QUIC] 1978 Marx, R., Ed., Niccolini, L., Ed., and M. Seemann, Ed., 1979 "QUIC event definitions for qlog", Work in Progress, 1980 Internet-Draft, draft-ietf-quic-qlog-quic-events-00, 1981 . 1984 11.2. Informative References 1986 [RFC1952] Deutsch, P., "GZIP file format specification version 4.3", 1987 RFC 1952, DOI 10.17487/RFC1952, May 1996, 1988 . 1990 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1991 Requirement Levels", BCP 14, RFC 2119, 1992 DOI 10.17487/RFC2119, March 1997, 1993 . 1995 [rfc4180] Shafranovich, Y., "Common Format and MIME Type for Comma- 1996 Separated Values (CSV) Files", RFC 4180, 1997 DOI 10.17487/RFC4180, October 2005, 1998 . 2000 [rfc7049] Bormann, C. and P. Hoffman, "Concise Binary Object 2001 Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, 2002 October 2013, . 2004 [RFC7932] Alakuijala, J. and Z. Szabadka, "Brotli Compressed Data 2005 Format", RFC 7932, DOI 10.17487/RFC7932, July 2016, 2006 . 2008 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 2009 Interchange Format", STD 90, RFC 8259, 2010 DOI 10.17487/RFC8259, December 2017, 2011 . 2013 Appendix A. Change Log 2015 A.1. Since draft-marx-qlog-main-schema-draft-02: 2017 * These changes were done in preparation of the adoption of the 2018 drafts by the QUIC working group (#137) 2020 * Moved RawInfo, Importance, Generic events and Simulation events to 2021 this document. 2023 * Added basic event definition guidelines 2025 * Made protocol_type an array instead of a string (#146) 2027 A.2. Since draft-marx-qlog-main-schema-01: 2029 * Decoupled qlog from the JSON format and described a mapping 2030 instead (#89) 2032 - Data types are now specified in this document and proper 2033 definitions for fields were added in this format 2035 - 64-bit numbers can now be either strings or numbers, with a 2036 preference for numbers (#10) 2038 - binary blobs are now logged as lowercase hex strings (#39, #36) 2040 - added guidance to add length-specifiers for binary blobs (#102) 2042 * Removed "time_units" from Configuration. All times are now in ms 2043 instead (#95) 2045 * Removed the "event_fields" setup for a more straightforward JSON 2046 format (#101,#89) 2048 * Added a streaming option using the NDJSON format (#109,#2,#106) 2050 * Described optional optimization options for implementers (#30) 2052 * Added QLOGDIR and QLOGFILE environment variables, clarified the 2053 .well-known URL usage (#26,#33,#51) 2055 * Overall tightened up the text and added more examples 2057 A.3. Since draft-marx-qlog-main-schema-00: 2059 * All field names are now lowercase (e.g., category instead of 2060 CATEGORY) 2062 * Triggers are now properties on the "data" field value, instead of 2063 separate field types (#23) 2065 * group_ids in common_fields is now just also group_id 2067 Appendix B. Design Variations 2069 * Quic-trace (https://github.com/google/quic-trace) takes a slightly 2070 different approach based on protocolbuffers. 2072 * Spindump (https://github.com/EricssonResearch/spindump) also 2073 defines a custom text-based format for in-network measurements 2075 * Wireshark (https://www.wireshark.org/) also has a QUIC dissector 2076 and its results can be transformed into a json output format using 2077 tshark. 2079 The idea is that qlog is able to encompass the use cases for both of 2080 these alternate designs and that all tooling converges on the qlog 2081 standard. 2083 Appendix C. Acknowledgements 2085 Much of the initial work by Robin Marx was done at Hasselt 2086 University. 2088 Thanks to Jana Iyengar, Brian Trammell, Dmitri Tikhonov, Stephen 2089 Petrides, Jari Arkko, Marcus Ihlar, Victor Vasiliev, Mirja 2090 Kuehlewind, Jeremy Laine and Lucas Pardue for their feedback and 2091 suggestions. 2093 Authors' Addresses 2095 Robin Marx (editor) 2096 KU Leuven 2098 Email: robin.marx@kuleuven.be 2100 Luca Niccolini (editor) 2101 Facebook 2103 Email: lniccolini@fb.com 2105 Marten Seemann (editor) 2106 Protocol Labs 2108 Email: marten@protocol.ai