idnits 2.17.00 (12 Aug 2021) /tmp/idnits50075/draft-ietf-trans-gossip-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 4 instances of lines with non-RFC2606-compliant FQDNs in the document. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 370: '...tension. The client MUST discard SCTs...' RFC 2119 keyword, line 371: '...own to the client and SHOULD store the...' RFC 2119 keyword, line 377: '...ed on the client MUST be keyed by the ...' RFC 2119 keyword, line 378: '...contacted. They MUST NOT be sent to t...' RFC 2119 keyword, line 381: '...le.com.) In particular, they MUST NOT...' (70 more instances...) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1841 has weird spacing: '... bool has_...' == Line 1842 has weird spacing: '... bool proo...' == Line 1948 has weird spacing: '... string doma...' -- The document date (January 14, 2018) is 1581 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Experimental ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 296 -- Looks like a reference, but probably isn't: '2' on line 298 -- Looks like a reference, but probably isn't: '3' on line 300 ** Obsolete normative reference: RFC 6962 (Obsoleted by RFC 9162) -- Duplicate reference: RFC6962, mentioned in 'RFC6962', was also mentioned in 'RFC-6962-BIS-27'. ** Obsolete normative reference: RFC 6962 (Obsoleted by RFC 9162) ** Obsolete normative reference: RFC 7159 (Obsoleted by RFC 8259) Summary: 4 errors (**), 0 flaws (~~), 5 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TRANS L. Nordberg 3 Internet-Draft NORDUnet 4 Intended status: Experimental D. Gillmor 5 Expires: July 18, 2018 ACLU 6 T. Ritter 7 January 14, 2018 9 Gossiping in CT 10 draft-ietf-trans-gossip-05 12 Abstract 14 The logs in Certificate Transparency are untrusted in the sense that 15 the users of the system don't have to trust that they behave 16 correctly since the behavior of a log can be verified to be correct. 18 This document tries to solve the problem with logs presenting a 19 "split view" of their operations or failing to incorporate a 20 submission within MMD. It describes three gossiping mechanisms for 21 Certificate Transparency: SCT Feedback, STH Pollination and Trusted 22 Auditor Relationship. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on July 18, 2018. 41 Copyright Notice 43 Copyright (c) 2018 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Defining the problem . . . . . . . . . . . . . . . . . . . . 4 60 3. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 62 4.1. Pre-Loaded vs Locally Added Anchors . . . . . . . . . . . 5 63 5. Who gossips with whom . . . . . . . . . . . . . . . . . . . . 5 64 6. What to gossip about and how . . . . . . . . . . . . . . . . 6 65 7. Data flow . . . . . . . . . . . . . . . . . . . . . . . . . . 6 66 8. Gossip Mechanisms . . . . . . . . . . . . . . . . . . . . . . 7 67 8.1. SCT Feedback . . . . . . . . . . . . . . . . . . . . . . 7 68 8.1.1. SCT Feedback data format . . . . . . . . . . . . . . 8 69 8.1.2. HTTPS client to server . . . . . . . . . . . . . . . 9 70 8.1.3. HTTPS server operation . . . . . . . . . . . . . . . 11 71 8.1.4. HTTPS server to auditors . . . . . . . . . . . . . . 13 72 8.2. STH pollination . . . . . . . . . . . . . . . . . . . . . 14 73 8.2.1. HTTPS Clients and Proof Fetching . . . . . . . . . . 16 74 8.2.2. STH Pollination without Proof Fetching . . . . . . . 17 75 8.2.3. Auditor Action . . . . . . . . . . . . . . . . . . . 17 76 8.2.4. STH Pollination data format . . . . . . . . . . . . . 18 77 8.3. Trusted Auditor Stream . . . . . . . . . . . . . . . . . 18 78 8.3.1. Trusted Auditor data format . . . . . . . . . . . . . 19 79 9. 3-Method Ecosystem . . . . . . . . . . . . . . . . . . . . . 20 80 9.1. SCT Feedback . . . . . . . . . . . . . . . . . . . . . . 20 81 9.2. STH Pollination . . . . . . . . . . . . . . . . . . . . . 20 82 9.3. Trusted Auditor Relationship . . . . . . . . . . . . . . 21 83 9.4. Interaction . . . . . . . . . . . . . . . . . . . . . . . 22 84 10. Security considerations . . . . . . . . . . . . . . . . . . . 23 85 10.1. Attacks by actively malicious logs . . . . . . . . . . . 23 86 10.2. Dual-CA Compromise . . . . . . . . . . . . . . . . . . . 23 87 10.3. Censorship/Blocking considerations . . . . . . . . . . . 24 88 10.4. Flushing Attacks . . . . . . . . . . . . . . . . . . . . 25 89 10.4.1. STHs . . . . . . . . . . . . . . . . . . . . . . . . 25 90 10.4.2. SCTs & Certificate Chains on HTTPS Servers . . . . . 26 91 10.4.3. SCTs & Certificate Chains on HTTPS Clients . . . . . 27 92 10.5. Privacy considerations . . . . . . . . . . . . . . . . . 27 93 10.5.1. Privacy and SCTs . . . . . . . . . . . . . . . . . . 27 94 10.5.2. Privacy in SCT Feedback . . . . . . . . . . . . . . 27 95 10.5.3. Privacy for HTTPS clients performing STH Proof 96 Fetching . . . . . . . . . . . . . . . . . . . . . . 28 98 10.5.4. Privacy in STH Pollination . . . . . . . . . . . . . 29 99 10.5.5. Privacy in STH Interaction . . . . . . . . . . . . . 29 100 10.5.6. Trusted Auditors for HTTPS Clients . . . . . . . . . 30 101 10.5.7. HTTPS Clients as Auditors . . . . . . . . . . . . . 30 102 11. Policy Recommendations . . . . . . . . . . . . . . . . . . . 31 103 11.1. Blocking Recommendations . . . . . . . . . . . . . . . . 31 104 11.1.1. Frustrating blocking . . . . . . . . . . . . . . . . 31 105 11.1.2. Responding to possible blocking . . . . . . . . . . 31 106 11.2. Proof Fetching Recommendations . . . . . . . . . . . . . 33 107 11.3. Record Distribution Recommendations . . . . . . . . . . 33 108 11.3.1. Mixing Algorithm . . . . . . . . . . . . . . . . . . 34 109 11.3.2. The Deletion Algorithm . . . . . . . . . . . . . . . 35 110 11.4. Concrete Recommendations . . . . . . . . . . . . . . . . 36 111 11.4.1. STH Pollination . . . . . . . . . . . . . . . . . . 36 112 11.4.2. SCT Feedback . . . . . . . . . . . . . . . . . . . . 40 113 12. IANA considerations . . . . . . . . . . . . . . . . . . . . . 53 114 13. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 53 115 14. ChangeLog . . . . . . . . . . . . . . . . . . . . . . . . . . 53 116 14.1. Changes between ietf-04 and ietf-05 . . . . . . . . . . 54 117 14.2. Changes between ietf-03 and ietf-04 . . . . . . . . . . 54 118 14.3. Changes between ietf-02 and ietf-03 . . . . . . . . . . 54 119 14.4. Changes between ietf-01 and ietf-02 . . . . . . . . . . 54 120 14.5. Changes between ietf-00 and ietf-01 . . . . . . . . . . 54 121 14.6. Changes between -01 and -02 . . . . . . . . . . . . . . 55 122 14.7. Changes between -00 and -01 . . . . . . . . . . . . . . 55 123 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 55 124 15.1. Normative References . . . . . . . . . . . . . . . . . . 55 125 15.2. Informative References . . . . . . . . . . . . . . . . . 56 126 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 57 128 1. Introduction 130 The purpose of the protocols in this document, collectively referred 131 to as CT Gossip, is to detect certain misbehavior by CT logs. In 132 particular, CT Gossip aims to detect logs that are providing 133 inconsistent views to different log clients, and logs failing to 134 include submitted certificates within the time period stipulated by 135 MMD. 137 One of the major challenges of any gossip protocol is limiting damage 138 to user privacy. The goal of CT gossip is to publish and distribute 139 information about the logs and their operations, but not to expose 140 any additional information about the operation of any of the other 141 participants. Privacy of consumers of log information (in 142 particular, of web browsers and other TLS clients) should not be 143 undermined by gossip. 145 This document presents three different, complementary mechanisms for 146 non-log elements of the CT ecosystem to exchange information about 147 logs in a manner that preserves the privacy of HTTPS clients. They 148 should provide protective benefits for the system as a whole even if 149 their adoption is not universal. 151 2. Defining the problem 153 When a log provides different views of the log to different clients 154 this is described as a partitioning attack. Each client would be 155 able to verify the append-only nature of the log but, in the extreme 156 case, each client might see a unique view of the log. 158 The CT logs are public, append-only and untrusted and thus have to be 159 audited for consistency, i.e., they should never rewrite history. 160 Additionally, auditors and other log clients need to exchange 161 information about logs in order to be able to detect a partitioning 162 attack (as described above). 164 Gossiping about log behavior helps address the problem of detecting 165 malicious or compromised logs with respect to a partitioning attack. 166 We want some side of the partitioned tree, and ideally all sides, to 167 see at least one other side. 169 Disseminating information about a log poses a potential threat to the 170 privacy of end users. Some data of interest (e.g., SCTs) is linkable 171 to specific log entries and thereby to specific websites, which makes 172 sharing them with others a privacy concern. Gossiping about this 173 data has to take privacy considerations into account in order not to 174 expose associations between users of the log (e.g., web browsers) and 175 certificate holders (e.g., web sites). Even sharing STHs (which do 176 not link to specific log entries) can be problematic - user tracking 177 by fingerprinting through rare STHs is one potential attack (see 178 Section 8.2). 180 3. Overview 182 This document presents three gossiping mechanisms: SCT Feedback, STH 183 Pollination, and a Trusted Auditor Relationship. 185 SCT Feedback enables HTTPS clients to share Signed Certificate 186 Timestamps (SCTs) (Section 4.8 of [RFC-6962-BIS-27]) with CT auditors 187 in a privacy-preserving manner by sending SCTs to originating HTTPS 188 servers, which in turn share them with CT auditors. 190 In STH Pollination, HTTPS clients use HTTPS servers as pools to share 191 Signed Tree Heads (STHs) (Section 4.10 of [RFC-6962-BIS-27]) with 192 other connecting clients in the hope that STHs will find their way to 193 CT auditors. 195 HTTPS clients in a Trusted Auditor Relationship share SCTs and STHs 196 with trusted CT auditors directly, with expectations of privacy 197 sensitive data being handled according to whatever privacy policy is 198 agreed on between client and trusted party. 200 Despite the privacy risks with sharing SCTs there is no loss in 201 privacy if a client sends SCTs for a given site to the site 202 corresponding to the SCT. This is because the site's cookies could 203 already indicate that the client had accessed that site. In this way 204 a site can accumulate records of SCTs that have been issued by 205 various logs for that site, providing a consolidated repository of 206 SCTs that could be shared with auditors. Auditors can use this 207 information to detect a misbehaving log that fails to include a 208 certificate within the time period stipulated by its MMD log 209 parameter. 211 Sharing an STH is considered reasonably safe from a privacy 212 perspective as long as the same STH is shared by a large number of 213 other log clients. This safety in numbers can be achieved by only 214 allowing gossiping of STHs issued in a certain window of time, while 215 also refusing to gossip about STHs from logs with too high an STH 216 issuance frequency (see Section 8.2). 218 4. Terminology 220 This document relies on terminology and data structures defined in 221 [RFC-6962-BIS-27], including MMD, STH, SCT, Version, LogID, SCT 222 timestamp, CtExtensions, SCT signature, Merkle Tree Hash. 224 This document relies on terminology defined in 225 [draft-ietf-trans-threat-analysis-12], including Auditing. 227 4.1. Pre-Loaded vs Locally Added Anchors 229 Through the document, we refer to both Trust Anchors (Certificate 230 Authorities) and Logs. Both Logs and Trust Anchors may be locally 231 added by an administrator. Unless otherwise clarified, in both cases 232 we refer to the set of Trust Anchors and Logs that come pre-loaded 233 and pre-trusted in a piece of client software. 235 5. Who gossips with whom 237 o HTTPS clients and servers (SCT Feedback and STH Pollination) 239 o HTTPS servers and CT auditors (SCT Feedback and STH Pollination) 240 o CT auditors (Trusted Auditor Relationship) 242 Additionally, some HTTPS clients may engage with an auditor which 243 they trust with their privacy: 245 o HTTPS clients and CT auditors (Trusted Auditor Relationship) 247 6. What to gossip about and how 249 There are three separate gossip streams: 251 o SCT Feedback - transporting SCTs and certificate chains from HTTPS 252 clients to CT auditors via HTTPS servers. 254 o STH Pollination - HTTPS clients and CT auditors using HTTPS 255 servers as STH pools for exchanging STHs. 257 o Trusted Auditor Stream - HTTPS clients communicating directly with 258 trusted CT auditors sharing SCTs, certificate chains and STHs. 260 It is worthwhile to note that when an HTTPS client or CT auditor 261 interacts with a log, they may equivalently interact with a log 262 mirror or cache that replicates the log. 264 7. Data flow 266 The following picture shows how certificates, SCTs and STHs flow 267 through a CT system with SCT Feedback and STH Pollination. It does 268 not show what goes in the Trusted Auditor Relationship stream. 270 +- Cert ---- +----------+ 271 | | CA | ----------+ 272 | + SCT -> +----------+ | 273 v | Cert [& SCT] 274 +----------+ | 275 | Log | ---------- SCT -----------+ 276 +----------+ v 277 | ^ +----------+ 278 | | SCTs & Certs --- | Website | 279 | |[1] | +----------+ 280 | |[2] STHs ^ | 281 | |[3] v | HTTPS traffic 282 | | +----------+ | | 283 | +--------> | Auditor | | SCT & Cert 284 | +----------+ | | 285 STH | STH & Inclusion proof 286 | | | 287 Log entries SCTs & Certs | 288 | | | 289 v STHs | 290 +----------+ | v 291 | Monitor | +----------+ 292 +----------+ | Browser | 293 +----------+ 295 # Auditor Log 296 [1] |--- get-sth ------------------->| 297 |<-- STH ------------------------| 298 [2] |--- leaf hash + tree size ----->| 299 |<-- index + inclusion proof --->| 300 [3] |--- tree size 1 + tree size 2 ->| 301 |<-- consistency proof ----------| 303 8. Gossip Mechanisms 305 8.1. SCT Feedback 307 The goal of SCT Feedback is for clients to share SCTs and certificate 308 chains with CT auditors while still preserving the privacy of the end 309 user. The sharing of SCTs contribute to the overall goal of 310 detecting misbehaving logs by providing auditors with SCTs from many 311 vantage points, making it more likely to catch a violation of a log's 312 MMD or a log presenting inconsistent views. The sharing of 313 certificate chains is beneficial to HTTPS server operators interested 314 in direct feedback from clients for detecting bogus certificates 315 issued in their name and therefore incentivizes server operators to 316 take part in SCT Feedback. 318 SCT Feedback is the most privacy-preserving gossip mechanism, as it 319 does not directly expose any links between an end user and the sites 320 they've visited to any third party. 322 HTTPS clients store SCTs and certificate chains they see, and later 323 send them to the originating HTTPS server by posting them to a well- 324 known URL (associated with that server), as described in 325 Section 8.1.2. Note that clients will send the same SCTs and chains 326 to a server multiple times with the assumption that any man-in-the- 327 middle attack eventually will cease, and an honest server will 328 eventually receive collected malicious SCTs and certificate chains. 330 HTTPS servers store SCTs and certificate chains received from 331 clients, as described in Section 8.1.3. They later share them with 332 CT auditors by either posting them to auditors or making them 333 available via a well-known URL. This is described in Section 8.1.4. 335 8.1.1. SCT Feedback data format 337 The data shared between HTTPS clients and servers, as well as between 338 HTTPS servers and CT auditors, is a JSON array [RFC7159]. Each item 339 in the array is a JSON object containing at least the first of the 340 following members: 342 o "x509_chain" : An array of PEM-encoded X.509 certificates. The 343 first element is the end-entity certificate, the second certifies 344 the first and so on. The "x509_chain" member is mandatory to 345 include. 347 o "sct_data_v1" : An array of base64 encoded 348 "SignedCertificateTimestampList"s as defined in [RFC6962] section 349 3.3. The "sct_data_v1" member is optional. 351 o "sct_data_v2" : An array of base64 encoded "TransItem" structures 352 of type "x509_sct_v2" or "precert_sct_v2" as defined in 353 [RFC-6962-BIS-27] section 4.8. The "sct_data_v2" member is 354 optional. 356 We will refer to this object as 'sct_feedback'. 358 The x509_chain element always contains a full chain from a leaf 359 certificate to a self-signed trust anchor. 361 See Section 8.1.2 for details on what the sct_data element contains 362 as well as more details about the x509_chain element. 364 8.1.2. HTTPS client to server 366 When an HTTPS client connects to an HTTPS server, the client receives 367 a set of SCTs as part of the TLS handshake. SCTs are included in the 368 TLS handshake using one or more of the three mechanisms described in 369 [RFC-6962-BIS-27] section 6 - in the server certificate, in a TLS 370 extension, or in an OCSP extension. The client MUST discard SCTs 371 that are not signed by a log known to the client and SHOULD store the 372 remaining SCTs together with a locally constructed certificate chain 373 which is trusted (i.e., terminated in a pre-loaded or locally 374 installed Trust Anchor) in an sct_feedback object or equivalent data 375 structure for later use in SCT Feedback. 377 The SCTs stored on the client MUST be keyed by the exact domain name 378 the client contacted. They MUST NOT be sent to the well-known URI of 379 any domain not matching the original domain (e.g., if the original 380 domain is sub.example.com they must not be sent to 381 sub.sub.example.com or to example.com.) In particular, they MUST NOT 382 be sent to the well-known URI of any Subject Alternate Names 383 specified in the certificate. In the case of certificates that 384 validate multiple domain names, after visiting a second domain name 385 specified in the certificate, the same SCT is expected to be stored 386 once under each domain name's key. If Connection Reuse as defined in 387 [RFC7540] is available, reusing an existing connection to 388 sub.example.com to send data to sub.sub.example.com is permitted. 390 Not following these constraints would increase the risk for two types 391 of privacy breaches. First, the HTTPS server receiving the SCT would 392 learn about other sites visited by the HTTPS client. Second, 393 auditors receiving SCTs from the HTTPS server would learn information 394 about other HTTPS servers visited by its clients. 396 If the client later again connects to the same HTTPS server, it again 397 receives a set of SCTs and calculates a certificate chain, and again 398 creates an sct_feedback or similar object. If this object does not 399 exactly match an existing object in the store, then the client MUST 400 add this new object to the store, associated with the exact domain 401 name contacted, as described above. An exact comparison is needed to 402 ensure that attacks involving alternate chains are detected. An 403 example of such an attack is described in 404 [dual-ca-compromise-attack]. However, at least one optimization is 405 safe and MAY be performed: If the certificate chain exactly matches 406 an existing certificate chain, the client MAY store the union of the 407 SCTs from the two objects in the first (existing) object. 409 If the client does connect to the same HTTPS server a subsequent 410 time, it MUST send to the server sct_feedback objects in the store 411 that are associated with that domain name. However, it is not 412 necessary to send an sct_feedback object constructed from the current 413 TLS session, and if the client does so, it MUST NOT be marked as sent 414 in any internal tracking done by the client. 416 Refer to Section 11.3 for recommendations for implementation. 418 Because SCTs can be used as a tracking mechanism (see 419 Section 10.5.2), they deserve special treatment when they are 420 received from (and provided to) domains that are loaded as 421 subresources from an origin domain. Such domains are commonly called 422 'third party domains'. An HTTPS client SHOULD store SCT Feedback 423 using a 'double-keying' approach, which isolates third party domains 424 by the first party domain. This is described in [double-keying]. 425 Gossip would be performed normally for third party domains only when 426 the user revisits the first party domain. In lieu of 'double- 427 keying', an HTTPS client MAY treat SCT Feedback in the same manner it 428 treats other security mechanisms that can enable tracking (such as 429 HSTS and HPKP.) 431 SCT Feedback is only performed when a user connects to a site via 432 intentional web browsing or normal third party resource inclusion. 433 It MUST NOT be performed automatically as part of some sort of 434 background process. 436 Finally, if the HTTPS client has configuration options for not 437 sending cookies to third parties, SCTs of third parties MUST be 438 treated as cookies with respect to this setting. This prevents third 439 party tracking through the use of SCTs/certificates, which would 440 bypass the cookie policy. For domains that are only loaded as third 441 party domains, the client may never perform SCT Feedback; however the 442 client may perform STH Pollination after fetching an inclusion proof, 443 as specified in Section 8.2. 445 SCTs and corresponding certificates are POSTed to the originating 446 HTTPS server at the well-known URL: 448 https:///.well-known/ct-gossip/v1/sct-feedback 450 The data sent in the POST is defined in Section 8.1.1. This data 451 SHOULD be sent in an already-established TLS session. This makes it 452 hard for an attacker to disrupt SCT Feedback without also disturbing 453 ordinary secure browsing (https://). This is discussed more in 454 Section 11.1.1. 456 The HTTPS server SHOULD respond with an HTTP 200 response code and an 457 empty body if it was able to process the request. An HTTPS client 458 which receives any other response SHOULD consider it an error. 460 Some clients have trust anchors or logs that are locally added (e.g., 461 by an administrator or by the user themselves). These additions are 462 potentially privacy-sensitive because they can carry information 463 about the specific configuration, computer, or user. 465 Certificates validated by locally added trust anchors will commonly 466 have no SCTs associated with them, so in this case no action is 467 needed with respect to CT Gossip. SCTs issued by locally added logs 468 MUST NOT be reported via SCT Feedback. 470 If a certificate is validated by SCTs that are issued by publicly 471 trusted logs, but chains to a local trust anchor, the client MAY 472 perform SCT Feedback for this SCT and certificate chain bundle. If 473 it does so, the client MUST include the full chain of certificates 474 chaining to the local trust anchor in the x509_chain array. 475 Performing SCT Feedback in this scenario may be advantageous for the 476 broader internet and CT ecosystem, but may also disclose information 477 about the client. If the client elects to omit SCT Feedback, it can 478 choose to perform STH Pollination after fetching an inclusion proof, 479 as specified in Section 8.2. 481 We require the client to send the full chain (or nothing at all) for 482 two reasons. Firstly, it simplifies the operation on the server if 483 there are not two code paths. Secondly, omitting the chain does not 484 actually preserve user privacy. The Issuer field in the certificate 485 describes the signing certificate. And if the certificate is being 486 submitted at all, it means the certificate is logged, and has SCTs. 487 This means that the Issuer can be queried and obtained from the log, 488 so omitting the signing certificate from the client's submission does 489 not actually help user privacy. 491 8.1.3. HTTPS server operation 493 HTTPS servers can be configured (or omit configuration), resulting 494 in, broadly, two modes of operation. In the simpler mode, the server 495 will only track leaf certificates and SCTs applicable to those leaf 496 certificates. In the more complex mode, the server will confirm the 497 client's chain validation and store the certificate chain. The 498 latter mode requires more configuration, but is necessary to prevent 499 denial of service (DoS) attacks on the server's storage space. 501 In the simple mode of operation, upon receiving a submission at the 502 sct-feedback well-known URL, an HTTPS server will perform a set of 503 operations, checking on each sct_feedback object before storing it: 505 o (1) the HTTPS server MAY modify the sct_feedback object, and 506 discard all items in the x509_chain array except the first item 507 (which is the end-entity certificate) 509 o (2) if a bit-wise compare of the sct_feedback object matches one 510 already in the store, this sct_feedback object SHOULD be discarded 512 o (3) if the leaf cert is not for a domain for which the server is 513 authoritative, the SCT MUST be discarded 515 o (4) if an SCT in the sct_data array can't be verified to be a 516 valid SCT for the accompanying leaf cert, and issued by a known 517 log, the individual SCT SHOULD be discarded 519 The modification in step number 1 is necessary to prevent a malicious 520 client from exhausting the server's storage space. A client can 521 generate their own issuing certificate authorities, and create an 522 arbitrary number of chains that terminate in an end-entity 523 certificate with an existing SCT. By discarding all but the end- 524 entity certificate, we prevent a simple HTTPS server from storing 525 this data. Note that operation in this mode will not prevent the 526 attack described in [dual-ca-compromise-attack]. Skipping this step 527 requires additional configuration as described below. 529 The check in step 2 is for detecting duplicates and minimizing 530 processing and storage by the server. As on the client, an exact 531 comparison is needed to ensure that attacks involving alternate 532 chains are detected. Again, at least one optimization is safe and 533 MAY be performed. If the certificate chain exactly matches an 534 existing certificate chain, the server MAY store the union of the 535 SCTs from the two objects in the first (existing) object. If the 536 validity check on any of the SCTs fails, the server SHOULD NOT store 537 the union of the SCTs. 539 The check in step 3 is to help malfunctioning clients from exposing 540 which sites they visit. It additionally helps prevent DoS attacks on 541 the server. 543 The check in step 4 is to prevent DoS attacks where an adversary 544 fills up the store prior to attacking a client (thus preventing the 545 client's feedback from being recorded), or an attack where an 546 adversary simply attempts to fill up server's storage space. 548 The above describes the simpler mode of operation. In the more 549 advanced server mode, the server will detect the attack described in 550 [dual-ca-compromise-attack]. In this configuration the server will 551 not modify the sct_feedback object prior to performing checks 2, 3, 552 and 4. Instead, to prevent a malicious client from filling the 553 server's data store, the HTTPS server SHOULD perform an additional 554 check in the more advanced mode: 556 o (5) if the x509_chain consists of an invalid certificate chain, or 557 the culminating trust anchor is not recognized by the server, the 558 server SHOULD modify the sct_feedback object, discarding all items 559 in the x509_chain array except the first item 561 The HTTPS server MAY choose to omit checks 4 or 5. This will place 562 the server at risk of having its data store filled up by invalid 563 data, but can also allow a server to identify interesting certificate 564 or certificate chains that omit valid SCTs, or do not chain to a 565 trusted root. This information may enable an HTTPS server operator 566 to detect attacks or unusual behavior of Certificate Authorities even 567 outside the Certificate Transparency ecosystem. 569 8.1.4. HTTPS server to auditors 571 HTTPS servers receiving SCTs from clients SHOULD share SCTs and 572 certificate chains with CT auditors by either serving them on the 573 well-known URL: 575 https:///.well-known/ct-gossip/v1/collected-sct-feedback 577 or by HTTPS POSTing them to a set of preconfigured auditors. This 578 allows an HTTPS server to choose between an active push model or a 579 passive pull model. 581 The data received in a GET of the well-known URL or sent in the POST 582 is defined in Section 8.1.1 with the following difference: The 583 x509_chain element may contain only he end-entity certificate, as 584 described below. 586 HTTPS servers SHOULD share all sct_feedback objects they see that 587 pass the checks in Section 8.1.3. If this is an infeasible amount of 588 data, the server MAY choose to expire submissions according to an 589 undefined policy. Suggestions for such a policy can be found in 590 Section 11.3. 592 HTTPS servers MUST NOT share any other data that they may learn from 593 the submission of SCT Feedback by HTTPS clients, like the HTTPS 594 client IP address or the time of submission. 596 As described above, HTTPS servers can be configured (or omit 597 configuration), resulting in two modes of operation. In one mode, 598 the x509_chain array will contain a full certificate chain. This 599 chain may terminate in a trust anchor the auditor may recognize, or 600 it may not. (One scenario where this could occur is if the client 601 submitted a chain terminating in a locally added trust anchor, and 602 the server kept this chain.) In the other mode, the x509_chain array 603 will consist of only a single element, which is the end-entity 604 certificate. 606 Auditors SHOULD provide the following URL accepting HTTPS POSTing of 607 SCT feedback data: 609 https:///ct-gossip/v1/sct-feedback 611 Auditors SHOULD regularly poll HTTPS servers at the well-known 612 collected-sct-feedback URL. The frequency of the polling and how to 613 determine which domains to poll is outside the scope of this 614 document. However, the selection MUST NOT be influenced by potential 615 HTTPS clients connecting directly to the auditor. For example, if a 616 poll to example.com occurs directly after a client submits an SCT for 617 example.com, an adversary observing the auditor can trivially 618 conclude the activity of the client. 620 8.2. STH pollination 622 The goal of sharing Signed Tree Heads (STHs) through pollination is 623 to share STHs between HTTPS clients and CT auditors while still 624 preserving the privacy of the end user. The sharing of STHs 625 contribute to the overall goal of detecting misbehaving logs by 626 providing CT auditors with STHs from many vantage points, making it 627 possible to detect logs that are presenting inconsistent views. 629 HTTPS servers supporting the protocol act as STH pools. HTTPS 630 clients and CT auditors in the possession of STHs can pollinate STH 631 pools by sending STHs to them, and retrieving new STHs to send to 632 other STH pools. CT auditors can improve the value of their auditing 633 by retrieving STHs from pools. 635 HTTPS clients send STHs to HTTPS servers by POSTing them to the well- 636 known URL: 638 https:///.well-known/ct-gossip/v1/sth-pollination 640 The data sent in the POST is defined in Section 8.2.4. This data 641 SHOULD be sent in an already established TLS session. This makes it 642 hard for an attacker to disrupt STH gossiping without also disturbing 643 ordinary secure browsing (https://). This is discussed more in 644 Section 11.1.1. 646 On a successful connection to an HTTPS server implementing STH 647 Pollination, the response code will be 200, and the response body is 648 application/json, containing zero or more STHs in the same format, as 649 described in Section 8.2.4. 651 An HTTPS client may acquire STHs by several methods: 653 o in replies to pollination POSTs; 655 o asking logs that it recognizes for the current STH, either 656 directly (v2/get-sth) or indirectly (for example over DNS) 658 o resolving an SCT and certificate to an STH via an inclusion proof 660 o resolving one STH to another via a consistency proof 662 HTTPS clients (that have STHs) and CT auditors SHOULD pollinate STH 663 pools with STHs. Which STHs to send and how often pollination should 664 happen is regarded as undefined policy with the exception of privacy 665 concerns explained below. Suggestions for the policy can be found in 666 Section 11.3. 668 An HTTPS client could be tracked by giving it a unique or rare STH. 669 To address this concern, we place restrictions on different 670 components of the system to ensure an STH will not be rare. 672 o HTTPS clients silently ignore STHs from logs with an STH issuance 673 frequency of more than one STH per hour. Logs use the STH 674 Frequency Count log parameter to express this ([RFC-6962-BIS-27] 675 section 4.1). 677 o HTTPS clients silently ignore STHs which are not fresh. 679 An STH is considered fresh iff its timestamp is less than 14 days in 680 the past. Given a maximum STH issuance rate of one per hour, an 681 attacker has 336 unique STHs per log for tracking. Clients MUST 682 ignore STHs older than 14 days. We consider STHs within this 683 validity window not to be personally identifiable data, and STHs 684 outside this window to be personally identifiable. 686 When multiplied by the number of logs from which a client accepts 687 STHs, this number of unique STHs grow and the negative privacy 688 implications grow with it. It's important that this is taken into 689 account when logs are chosen for default settings in HTTPS clients. 690 This concern is discussed upon in Section 10.5.5. 692 A log may cease operation, in which case there will soon be no STH 693 within the validity window. Clients SHOULD perform all three methods 694 of gossip about a log that has ceased operation since it is possible 695 the log was still compromised and gossip can detect that. STH 696 Pollination is the one mechanism where a client must know about a log 697 shutdown. A client which does not know about a log shutdown MUST NOT 698 attempt any heuristic to detect a shutdown. Instead the client MUST 699 be informed about the shutdown from a verifiable source (e.g., a 700 software update), and be provided the final STH issued by the log. 701 The client SHOULD resolve SCTs and STHs to this final STH. If an SCT 702 or STH cannot be resolved to the final STH, clients SHOULD follow the 703 requirements and recommendations set forth in Section 11.1.2. 705 8.2.1. HTTPS Clients and Proof Fetching 707 There are two types of proofs a client may retrieve; inclusion proofs 708 and consistency proofs. 710 An HTTPS client will retrieve SCTs together with certificate chains 711 from an HTTPS server. Using the timestamp in the SCT together with 712 the end-entity certificate and the issuer key hash, it can obtain an 713 inclusion proof to an STH in order to verify the promise made by the 714 SCT. 716 An HTTPS client will have STHs from performing STH Pollination, and 717 may obtain a consistency proof to a more recent STH. 719 An HTTPS client may also receive an SCT bundled with an inclusion 720 proof to a historical STH via an unspecified future mechanism. 721 Because this historical STH is considered personally identifiable 722 information per above, the client needs to obtain a consistency proof 723 to a more recent STH. 725 A client SHOULD attempt proof fetching. A client MAY do network 726 probing to determine if proof fetching may succeed, and if it learns 727 that it does not, SHOULD periodically re-probe (especially after 728 network change, if it is aware of these events.) If it does succeed, 729 queued events can be processed. 731 A client MUST NOT perform proof fetching for any SCTs or STHs issued 732 by a locally added log. A client MAY fetch an inclusion proof for an 733 SCT (issued by a pre-loaded log) that validates a certificate 734 chaining to a locally added trust anchor. 736 If a client requested either proof directly from a log or auditor, it 737 would reveal the client's browsing habits to a third party. To 738 mitigate this risk, an HTTPS client MUST retrieve the proof in a 739 manner that disguises the client. 741 Depending on the client's DNS provider, DNS may provide an 742 appropriate intermediate layer that obfuscates the linkability 743 between the user of the client and the request for inclusion (while 744 at the same time providing a caching layer for oft-requested 745 inclusion proofs). See [draft-ct-over-dns] for an example of how 746 this can be done. 748 Anonymity networks such as Tor also present a mechanism for a client 749 to anonymously retrieve a proof from an auditor or log. 751 Even when using a privacy-preserving layer between the client and the 752 log, certain observations may be made about an anonymous client or 753 general user behavior depending on how proofs are fetched. For 754 example, if a client fetched all outstanding proofs at once, a log 755 would know that SCTs or STHs received around the same time are more 756 likely to come from a particular client. This could potentially go 757 so far as correlation of activity at different times to a single 758 client. In aggregate the data could reveal what sites are commonly 759 visited together. HTTPS clients SHOULD use a strategy of proof 760 fetching that attempts to obfuscate these patterns. A suggestion of 761 such a policy can be found in Section 11.2. 763 Resolving either SCTs and STHs may result in errors. These errors 764 may be routine downtime or other transient errors, or they may be 765 indicative of an attack. Clients SHOULD follow the requirements and 766 recommendations set forth in Section 11.1.2 when handling these 767 errors in order to give the CT ecosystem the greatest chance of 768 detecting and responding to a compromise. 770 8.2.2. STH Pollination without Proof Fetching 772 An HTTPS client MAY participate in STH Pollination without fetching 773 proofs. In this situation, the client receives STHs from a server, 774 applies the same validation logic to them (signed by a known log, 775 within the validity window) and will later pass them to another HTTPS 776 server. 778 When operating in this fashion, the HTTPS client is promoting gossip 779 for Certificate Transparency, but derives no direct benefit itself. 780 In comparison, a client which resolves SCTs or historical STHs to 781 recent STHs and pollinates them is assured that if it was attacked, 782 there is a probability that the ecosystem will detect and respond to 783 the attack (by distrusting the log). 785 8.2.3. Auditor Action 787 CT auditors participate in STH pollination by retrieving STHs from 788 HTTPS servers. They verify that the STH is valid by checking the 789 signature, and requesting a consistency proof from the STH to the 790 most recent STH. 792 After retrieving the consistency proof to the most recent STH, they 793 SHOULD pollinate this new STH among participating HTTPS servers. In 794 this way, as STHs "age out" and are no longer fresh, their "lineage" 795 continues to be tracked in the system. 797 8.2.4. STH Pollination data format 799 The data sent from HTTPS clients and CT auditors to HTTPS servers is 800 a JSON object [RFC7159] with one or both of the following two 801 members: 803 o "v1" : array of 0 or more objects each containing an STH as 804 returned from ct/v1/get-sth, see [RFC6962] section 4.3 806 o "v2" : array of 0 or more objects each containing an STH as 807 returned from ct/v2/get-sth, see [RFC-6962-BIS-27] section 5.2 809 Note that all STHs MUST be fresh as defined in Section 8.2. 811 8.3. Trusted Auditor Stream 813 HTTPS clients MAY send SCTs and cert chains, as well as STHs, 814 directly to auditors. If sent, this data MAY include data that 815 reflects locally added logs or trust anchors. Note that there are 816 privacy implications in doing so, these are outlined in 817 Section 10.5.1 and Section 10.5.6. 819 The most natural trusted auditor arrangement arguably is a web 820 browser that is "logged in to" a provider of various internet 821 services. Another equivalent arrangement is a trusted party like a 822 corporation to which an employee is connected through a VPN or by 823 other similar means. A third might be individuals or smaller groups 824 of people running their own services. In such a setting, retrieving 825 proofs from that third party could be considered reasonable from a 826 privacy perspective. The HTTPS client may also do its own auditing 827 and might additionally share SCTs and STHs with the trusted party to 828 contribute to herd immunity. Here, the ordinary [RFC-6962-BIS-27] 829 protocol is sufficient for the client to do the auditing while SCT 830 Feedback and STH Pollination can be used in whole or in parts for the 831 gossip part. 833 Another well established trusted party arrangement on the internet 834 today is the relation between internet users and their providers of 835 DNS resolver services. DNS resolvers are typically provided by the 836 internet service provider (ISP) used, which by the nature of name 837 resolving already know a great deal about which sites their users 838 visit. As mentioned in Section 8.2.1, in order for HTTPS clients to 839 be able to retrieve proofs in a privacy preserving manner, logs could 840 expose a DNS interface in addition to the ordinary HTTPS interface. 841 A specification of such a protocol can be found in 842 [draft-ct-over-dns]. 844 8.3.1. Trusted Auditor data format 846 Trusted Auditors expose a REST API at the fixed URI: 848 https:///ct-gossip/v1/trusted-auditor 850 Submissions are made by sending an HTTPS POST request, with the body 851 of the POST in a JSON object. Upon successful receipt the Trusted 852 Auditor returns 200 OK. 854 The JSON object consists of two top-level keys: 'sct_feedback' and 855 'sths'. The 'sct_feedback' value is an array of JSON objects as 856 defined in Section 8.1.1. The 'sths' value is an array of STHs as 857 defined in Section 8.2.4. 859 Example: 861 { 862 'sct_feedback' : 863 [ 864 { 865 'x509_chain' : 866 [ 867 '----BEGIN CERTIFICATE---\n 868 AAA...', 869 '----BEGIN CERTIFICATE---\n 870 AAA...', 871 ... 872 ], 873 'sct_data' : 874 [ 875 'AAA...', 876 'AAA...', 877 ... 878 ] 879 }, ... 880 ], 881 'sths' : 882 [ 883 'AAA...', 884 'AAA...', 885 ... 886 ] 887 } 889 9. 3-Method Ecosystem 891 The use of three distinct methods for auditing logs may seem 892 excessive, but each represents a needed component in the CT 893 ecosystem. To understand why, the drawbacks of each component must 894 be outlined. In this discussion we assume that an attacker knows 895 which mechanisms an HTTPS client and HTTPS server implement. 897 9.1. SCT Feedback 899 SCT Feedback requires the cooperation of HTTPS clients and more 900 importantly HTTPS servers. Although SCT Feedback does require a 901 significant amount of server-side logic to respond to the 902 corresponding APIs, this functionality does not require 903 customization, so it may be pre-provided and work out of the box. 904 However, to take full advantage of the system, an HTTPS server would 905 wish to perform some configuration to optimize its operation: 907 o Minimize its disk commitment by maintaining a list of known SCTs 908 and certificate chains (or hashes thereof) 910 o Maximize its chance of detecting a misissued certificate by 911 configuring a trust store of CAs 913 o Establish a "push" mechanism for POSTing SCTs to CT auditors 915 These configuration needs, and the simple fact that it would require 916 some deployment of software, means that some percentage of HTTPS 917 servers will not deploy SCT Feedback. 919 If SCT Feedback was the only mechanism in the ecosystem, any server 920 that did not implement the feature would open itself and its users to 921 attack without any possibility of detection. 923 A webserver not deploying SCT Feedback (or an alternative method 924 providing equivalent functionality) may never learn that it was a 925 target of an attack by a malicious log, as described in Section 10.1, 926 although the presence of an attack by the log could be learned 927 through STH Pollination. Additionally, users who wish to have the 928 strongest measure of privacy protection (by disabling STH Pollination 929 Proof Fetching and forgoing a Trusted Auditor) could be attacked 930 without risk of detection. 932 9.2. STH Pollination 934 STH Pollination requires the cooperation of HTTPS clients, HTTPS 935 servers, and logs. 937 For a client to fully participate in STH Pollination, and have this 938 mechanism detect attacks against it, the client must have a way to 939 safely perform Proof Fetching in a privacy preserving manner. (The 940 client may pollinate STHs it receives without performing Proof 941 Fetching, but we do not consider this option in this section.) 943 HTTPS servers must deploy software (although, as in the case with SCT 944 Feedback this logic can be pre-provided) and commit some configurable 945 amount of disk space to the endeavor. 947 Logs (or a third party mirroring the logs) must provide access to 948 clients to query proofs in a privacy preserving manner, most likely 949 through DNS. 951 Unlike SCT Feedback, the STH Pollination mechanism is not hampered if 952 only a minority of HTTPS servers deploy it. However, it makes an 953 assumption that an HTTPS client performs Proof Fetching (such as the 954 DNS mechanism discussed). Unfortunately, any manner that is 955 anonymous for some (such as clients which use shared DNS services 956 such as a large ISP), may not be anonymous for others. 958 For instance, DNS requests expose a considerable amount of sensitive 959 information (including what data is already present in the cache) in 960 plaintext over the network. For this reason, some percentage of 961 HTTPS clients may choose to not enable the Proof Fetching component 962 of STH Pollination. (Although they can still request and send STHs 963 among participating HTTPS servers, even when this affords them no 964 direct benefit.) 966 If STH Pollination was the only mechanism deployed, users that 967 disable it would be able to be attacked without risk of detection. 969 If STH Pollination (or an alternative method providing equivalent 970 functionality) was not deployed, HTTPS clients visiting HTTPS Servers 971 which did not deploy SCT Feedback could be attacked without risk of 972 detection. 974 9.3. Trusted Auditor Relationship 976 The Trusted Auditor Relationship is expected to be the rarest gossip 977 mechanism, as an HTTPS client is providing an unadulterated report of 978 its browsing history to a third party. While there are valid and 979 common reasons for doing so, there is no appropriate way to enter 980 into this relationship without retrieving informed consent from the 981 user. 983 However, the Trusted Auditor Relationship mechanism still provides 984 value to a class of HTTPS clients. For example, web crawlers have no 985 concept of a "user" and no expectation of privacy. Organizations 986 already performing network auditing for anomalies or attacks can run 987 their own Trusted Auditor for the same purpose with marginal increase 988 in privacy concerns. 990 The ability to change one's Trusted Auditor is a form of Trust 991 Agility that allows a user to choose who to trust, and be able to 992 revise that decision later without consequence. A Trusted Auditor 993 connection can be made more confidential than DNS (through the use of 994 TLS), and can even be made (somewhat) anonymous through the use of 995 anonymity services such as Tor. (Note that this does ignore the de- 996 anonymization possibilities available from viewing a user's browsing 997 history.) 999 If the Trusted Auditor relationship was the only mechanism deployed, 1000 users who do not enable it (the majority) would be able to be 1001 attacked without risk of detection. 1003 If the Trusted Auditor relationship was not deployed, crawlers and 1004 organizations would build it themselves for their own needs. By 1005 standardizing it, users who wish to opt-in (for instance those 1006 unwilling to participate fully in STH Pollination) can have an 1007 interoperable standard they can use to choose and change their 1008 trusted auditor. 1010 9.4. Interaction 1012 Assuming no other log consistency measures exist, clients who perform 1013 only a subset of the mechanisms described in this document are 1014 exposed to the following vulnerabilities: 1016 HTTPS clients can be attacked without risk of detection if they do 1017 not participate in any of the three mechanisms. 1019 HTTPS clients are afforded the greatest chance of detecting an attack 1020 when they either participate in both SCT Feedback and STH Pollination 1021 with Proof Fetching or if they have a Trusted Auditor relationship. 1022 (Participating in SCT Feedback is the only way specified in this 1023 document to prevent a malicious log from refusing to ever resolve an 1024 SCT to an STH, as put forward in Section 10.1). Additionally, 1025 participating in SCT Feedback enables an HTTPS client to assist in 1026 detecting the exact target of an attack. 1028 HTTPS servers that omit SCT Feedback enable malicious logs to carry 1029 out attacks without risk of detection. If these servers are targeted 1030 specifically, even if the attack is detected, without SCT Feedback 1031 they may never learn that they were specifically targeted. HTTPS 1032 servers without SCT Feedback do gain some measure of herd immunity, 1033 but only because their clients participate in STH Pollination (with 1034 Proof Fetching) or have a Trusted Auditor Relationship. 1036 When HTTPS servers omit SCT feedback, it allows their users to be 1037 attacked without detection by a malicious log; the vulnerable users 1038 are those who do not have a Trusted Auditor relationship. 1040 10. Security considerations 1042 10.1. Attacks by actively malicious logs 1044 One of the most powerful attacks possible in the CT ecosystem is a 1045 trusted log that has actively decided to be malicious. It can carry 1046 out an attack in at least two ways: 1048 In the first attack, the log can present a split view of the log for 1049 all time. This attack can be detected by CT auditors, but a naive 1050 auditor implementation may fail to do so. The simplest, least 1051 efficient way to detect the attack is to mirror the entire log and 1052 assert inclusion of every peice of data. If an auditor does not 1053 mirror the log, one way to detect this attack is to resolve each view 1054 of the log to the most recent STHs available and then force the log 1055 to present a consistency proof. (Which it cannot.) We highly 1056 recommend auditors plan for this attack scenario and ensure it will 1057 be detected. 1059 In the second attack, the log can sign an SCT, and refuse to ever 1060 include the certificate that the SCT refers to in the tree. 1061 (Alternately, it can include it in a branch of the tree and issue an 1062 STH, but then abandon that branch.) Whenever someone requests an 1063 inclusion proof for that SCT (or a consistency proof from that STH), 1064 the log would respond with an error, and a client may simply regard 1065 the response as a transient error. This attack can be detected using 1066 SCT Feedback, or an Auditor of Last Resort, as presented in 1067 Section 11.1.2. 1069 Both of these attack variants can be detected by CT auditors who have 1070 obtained an STH of an 'abnormal' view of the log. However, they may 1071 not be able to link the STH to any particular SCT or Certificate. 1072 This means that while the log misbehavior was successfully detected, 1073 the target of the attack was not identified. To assertively identify 1074 the target(s) of the attack, SCT Feedback is necessary. 1076 10.2. Dual-CA Compromise 1078 [dual-ca-compromise-attack] describes an attack possible by an 1079 adversary who compromises two Certificate Authorities and a Log. This 1080 attack is difficult to defend against in the CT ecosystem, and 1082 [dual-ca-compromise-attack] describes a few approaches to doing so. 1083 We note that Gossip is not intended to defend against this attack, 1084 but can in certain modes. 1086 Defending against the Dual-CA Compromise attack requires SCT 1087 Feedback, and explicitly requires the server to save full certificate 1088 chains (described in Section 8.1.3 as the 'complex' configuration.) 1089 After CT auditors receive the full certificate chains from servers, 1090 they MAY compare the chain built by clients to the chain supplied by 1091 the log. If the chains differ significantly, the auditor SHOULD 1092 raise a concern. A method of determining if chains differ 1093 significantly is by asserting that one chain is not a subset of the 1094 other and that the roots of the chains are different. 1096 10.3. Censorship/Blocking considerations 1098 We assume a network attacker who is able to fully control the 1099 client's internet connection for some period of time, including 1100 selectively blocking requests to certain hosts and truncating TLS 1101 connections based on information observed or guessed about client 1102 behavior. In order to successfully detect log misbehavior, the 1103 gossip mechanisms must still work even in these conditions. 1105 There are several gossip connections that can be blocked: 1107 1. Clients sending SCTs to servers in SCT Feedback 1109 2. Servers sending SCTs to auditors in SCT Feedback (server push 1110 mechanism) 1112 3. Servers making SCTs available to auditors (auditor pull 1113 mechanism) 1115 4. Clients fetching proofs in STH Pollination 1117 5. Clients sending STHs to servers in STH Pollination 1119 6. Servers sending STHs to clients in STH Pollination 1121 7. Clients sending SCTs to Trusted Auditors 1123 If a party cannot connect to another party, it can be assured that 1124 the connection did not succeed. While it may not have been 1125 maliciously blocked, it knows the transaction did not succeed. 1126 Mechanisms which result in a positive affirmation from the recipient 1127 that the transaction succeeded allow confirmation that a connection 1128 was not blocked. In this situation, the party can factor this into 1129 strategies suggested in Section 11.3 and in Section 11.1.2. 1131 The connections that allow positive affirmation are 1, 2, 4, 5, and 1132 7. 1134 More insidious is blocking the connections that do not allow positive 1135 confirmation: 3 and 6. An attacker may truncate or drop a response 1136 from a server to a client, such that the server believes it has 1137 shared data with the recipient, when it has not. However, in both 1138 scenarios (3 and 6), the server cannot distinguish the client as a 1139 cooperating member of the CT ecosystem or as an attacker performing a 1140 Sybil attack, aiming to flush the server's data store. Therefore the 1141 fact that these connections can be undetectably blocked does not 1142 actually alter the threat model of servers responding to these 1143 requests. The choice of algorithm to release data is crucial to 1144 protect against these attacks; strategies are suggested in 1145 Section 11.3. 1147 Handling censorship and network blocking (which is indistinguishable 1148 from network error) is relegated to the implementation policy chosen 1149 by clients. Suggestions for client behavior are specified in 1150 Section 11.1. 1152 10.4. Flushing Attacks 1154 A flushing attack is an attempt by an adversary to flush a particular 1155 piece of data from a pool. In the CT Gossip ecosystem, an attacker 1156 may have performed an attack and left evidence of a compromised log 1157 on a client or server. They would be interested in flushing that 1158 data, i.e. tricking the target into gossiping or pollinating the 1159 incriminating evidence with only attacker-controlled clients or 1160 servers with the hope they trick the target into deleting it. 1162 Flushing attacks may be defended against differently depending on the 1163 entity (HTTPS client or HTTPS server) and record (STHs or SCTs with 1164 Certificate Chains). 1166 10.4.1. STHs 1168 For both HTTPS clients and HTTPS servers, STHs within the validity 1169 window SHOULD NOT be deleted. An attacker cannot flush an item from 1170 the cache if it is never removed so flushing attacks are completely 1171 mitigated. 1173 The required disk space for all STHs within the validity window is 1174 336 STHs per log that is trusted. If 20 logs are trusted, and each 1175 STH takes 1 Kilobytes, this is 6.56 Megabytes. 1177 Note that it is important that implementors do not calculate the 1178 exact size of cache expected - if an attack does occur, a small 1179 number of additional, fraudulent STHs will enter into the cache. 1180 These STHs will be in addition to the expected set, and will be 1181 evidence of the attack. Flooding the cache will not work, as an 1182 attacker would have to include fraudulent STHs in the flood. 1184 If an HTTPS client or HTTPS server is operating in a constrained 1185 environment and cannot devote enough storage space to hold all STHs 1186 within the validity window it is recommended to use the below 1187 Deletion Algorithm in section Section 11.3.2 to make it more 1188 difficult for the attacker to perform a flushing attack. 1190 10.4.2. SCTs & Certificate Chains on HTTPS Servers 1192 An HTTPS server will only accept SCTs and Certificate Chains for 1193 domains it is authoritative for. Therefore the storage space needed 1194 is bound by the number of logs it accepts, multiplied by the number 1195 of domains it is authoritative for, multiplied by the number of 1196 certificates issued for those domains. 1198 Imagine a server authoritative for 10,000 domains, and each domain 1199 has 3 certificate chains, and 10 SCTs. A certificate chain is 5 1200 Kilobytes in size and an SCT 1 Kilobyte. This yields 732 Megabytes. 1202 This data can be large, but it is calculable. Web properties with 1203 more certificates and domains are more likely to be able to handle 1204 the increased storage need, while small web properties will not seen 1205 an undue burden. Therefore HTTPS servers SHOULD NOT delete SCTs or 1206 Certificate Chains. This completely mitigates flushing attacks. 1208 Again, note that it is important that implementors do not calculate 1209 the exact size of cache expected - if an attack does occur, the new 1210 SCT(s) and Certificate Chain(s) will enter into the cache. This data 1211 will be in addition to the expected set, and will be evidence of the 1212 attack. 1214 If an HTTPS server is operating in a constrained environment and 1215 cannot devote enough storage space to hold all SCTs and Certificate 1216 Chains it is authoritative for it is recommended to configure the SCT 1217 Feedback mechanism to allow only certain certificates that are known 1218 to be valid. These chains and SCTs can then be discarded without 1219 being stored or subsequently provided to any clients or auditors. If 1220 the allowlist is not sufficient, the below Deletion Algorithm in 1221 Section 11.3.2 is recommended to make it more difficult for the 1222 attacker to perform a flushing attack. 1224 10.4.3. SCTs & Certificate Chains on HTTPS Clients 1226 HTTPS clients will accumulate SCTs and Certificate Chains without 1227 bound. It is expected they will choose a particular cache size and 1228 delete entries when the cache size meets its limit. This does not 1229 mitigate flushing attacks, and such an attack is documented in 1230 [gossip-mixing]. 1232 The below Deletion Algorithm Section 11.3.2 is recommended to make it 1233 more difficult for the attacker to perform a flushing attack. 1235 10.5. Privacy considerations 1237 CT Gossip deals with HTTPS clients which are trying to share 1238 indicators that correspond to their browsing history. The most 1239 sensitive relationships in the CT ecosystem are the relationships 1240 between HTTPS clients and HTTPS servers. Client-server relationships 1241 can be aggregated into a network graph with potentially serious 1242 implications for correlative de-anonymization of clients and 1243 relationship-mapping or clustering of servers or of clients. 1245 There are, however, certain clients that do not require privacy 1246 protection. Examples of these clients are web crawlers or robots. 1247 But even in this case, the method by which these clients crawl the 1248 web may in fact be considered sensitive information. In general, it 1249 is better to err on the side of safety, and not assume a client is 1250 okay with giving up its privacy. 1252 10.5.1. Privacy and SCTs 1254 An SCT contains information that links it to a particular web site. 1255 Because the client-server relationship is sensitive, gossip between 1256 clients and servers about unrelated SCTs is risky. Therefore, a 1257 client with an SCT for a given server SHOULD NOT transmit that 1258 information in any other than the following two channels: to the 1259 server associated with the SCT itself (via a TLS connection with a 1260 certificate identifying the Domain Name of the web site with a Host 1261 header specifying the domain name); or to a Trusted Auditor, if one 1262 exists. 1264 10.5.2. Privacy in SCT Feedback 1266 SCTs introduce yet another mechanism for HTTPS servers to store state 1267 on an HTTPS client, and potentially track users. HTTPS clients which 1268 allow users to clear history or cookies associated with an origin 1269 MUST clear stored SCTs and certificate chains associated with the 1270 origin as well. 1272 Auditors should treat all SCTs as sensitive data. SCTs received 1273 directly from an HTTPS client are especially sensitive, because the 1274 auditor is a trusted by the client to not reveal their associations 1275 with servers. Auditors MUST NOT share such SCTs in any way, 1276 including sending them to an external log, without first mixing them 1277 with multiple other SCTs learned through submissions from multiple 1278 other clients. Suggestions for mixing SCTs are presented in 1279 Section 11.3. 1281 There is a possible fingerprinting attack where a log issues a unique 1282 SCT for targeted log client(s). A colluding log and HTTPS server 1283 operator could therefore be a threat to the privacy of an HTTPS 1284 client. Given all the other opportunities for HTTPS servers to 1285 fingerprint clients - TLS session tickets, HPKP and HSTS headers, 1286 HTTP Cookies, etc. - this is considered acceptable. 1288 The fingerprinting attack described above would be mitigated by a 1289 requirement that logs must use a deterministic signature scheme when 1290 signing SCTs ([RFC-6962-BIS-27] section 2.2). A log signing using 1291 RSA is not required to use a deterministic signature scheme. 1293 Since logs are allowed to issue a new SCT for a certificate already 1294 present in the log, mandating deterministic signatures does not stop 1295 this fingerprinting attack altogether. It does make the attack 1296 harder to pull off without being detected though. 1298 There is another similar fingerprinting attack where an HTTPS server 1299 tracks a client by using a unique certificate or a variation of cert 1300 chains. The risk for this attack is accepted on the same grounds as 1301 the unique SCT attack described above. 1303 10.5.3. Privacy for HTTPS clients performing STH Proof Fetching 1305 An HTTPS client performing Proof Fetching SHOULD NOT request proofs 1306 from a CT log that it doesn't accept SCTs from. An HTTPS client 1307 SHOULD regularly request an STH from all logs it is willing to 1308 accept, even if it has seen no SCTs from that log. 1310 The time between two polls for new STH's SHOULD NOT be significantly 1311 shorter than the MMD of the polled log divided by its STH Frequency 1312 Count ([RFC-6962-BIS-27] section 4.1). 1314 The actual mechanism by which Proof Fetching is done carries 1315 considerable privacy concerns. Although out of scope for the 1316 document, DNS is a mechanism currently discussed. DNS exposes data 1317 in plaintext over the network (including what sites the user is 1318 visiting and what sites they have previously visited) and may not be 1319 suitable for some. 1321 10.5.4. Privacy in STH Pollination 1323 An STH linked to an HTTPS client may indicate the following about 1324 that client: 1326 o that the client gossips; 1328 o that the client has been using CT at least until the time that the 1329 timestamp and the tree size indicate; 1331 o that the client is talking, possibly indirectly, to the log 1332 indicated by the tree hash; 1334 o which software and software version is being used. 1336 There is a possible fingerprinting attack where a log issues a unique 1337 STH for a targeted HTTPS client. This is similar to the 1338 fingerprinting attack described in Section 10.5.2, but can operate 1339 cross-origin. If a log (or HTTPS server cooperating with a log) 1340 provides a unique STH to a client, the targeted client will be the 1341 only client pollinating that STH cross-origin. 1343 It is mitigated partially because the log is limited in the number of 1344 STHs it can issue. It must 'save' one of its STHs each MMD to 1345 perform the attack. A log violating its STH Frequency Count 1346 ([RFC-6962-BIS-27] section 4.1) can be identified as non-compliant by 1347 CT auditors following the procedure described in [RFC-6962-BIS-27] 1348 section 8.3. 1350 10.5.5. Privacy in STH Interaction 1352 An HTTPS client may pollinate any STH within the last 14 days. An 1353 HTTPS client may also pollinate an STH for any log that it knows 1354 about. When a client pollinates STHs to a server, it will release 1355 more than one STH at a time. It is unclear if a server may 'prime' a 1356 client and be able to reliably detect the client at a later time. 1358 It's clear that a single site can track a user any way they wish, but 1359 this attack works cross-origin and is therefore more concerning. Two 1360 independent sites A and B want to collaborate to track a user cross- 1361 origin. A feeds a client Carol some N specific STHs from the M logs 1362 Carol trusts, chosen to be older and less common, but still in the 1363 validity window. Carol visits B and chooses to release some of the 1364 STHs she has stored, according to some policy. 1366 Modeling a representation for how common older STHs are in the pools 1367 of clients, and examining that with a given policy of how to choose 1368 which of those STHs to send to B, it should be possible to calculate 1369 statistics about how unique Carol looks when talking to B and how 1370 useful/accurate such a tracking mechanism is. 1372 Building such a model is likely impossible without some real world 1373 data, and requires a given implementation of a policy. To combat 1374 this attack, suggestions are provided in Section 11.3 to attempt to 1375 minimize it, but follow-up testing with real world deployment to 1376 improve the policy will be required. 1378 10.5.6. Trusted Auditors for HTTPS Clients 1380 Some HTTPS clients may choose to use a trusted auditor. This trust 1381 relationship exposes a large amount of information about the client 1382 to the auditor. In particular, it will identify the web sites that 1383 the client has visited to the auditor. Some clients may already 1384 share this information to a third party, for example, when using a 1385 server to synchronize browser history across devices in a server- 1386 visible way, or when doing DNS lookups through a trusted DNS 1387 resolver. For clients with such a relationship already established, 1388 sending SCTs to a trusted auditor run by the same organization does 1389 not appear to expose any additional information to the trusted third 1390 party. 1392 Clients which wish to contact a CT auditor without associating their 1393 identities with their SCTs may wish to use an anonymizing network 1394 like Tor to submit SCT Feedback to the auditor. Auditors SHOULD 1395 accept SCT Feedback that arrives over such anonymizing networks. 1397 Clients sending feedback to an auditor may prefer to reduce the 1398 temporal granularity of the history exposure to the auditor by 1399 caching and delaying their SCT Feedback reports. This is elaborated 1400 upon in Section 11.3. This strategy is only as effective as the 1401 granularity of the timestamps embedded in the SCTs and STHs. 1403 10.5.7. HTTPS Clients as Auditors 1405 Some HTTPS clients may choose to act as CT auditors themselves. A 1406 Client taking on this role needs to consider the following: 1408 o an Auditing HTTPS client potentially exposes its history to the 1409 logs that they query. Querying the log through a cache or a proxy 1410 with many other users may avoid this exposure, but may expose 1411 information to the cache or proxy, in the same way that a non- 1412 Auditing HTTPS Client exposes information to a Trusted Auditor. 1414 o an effective CT auditor needs a strategy about what to do in the 1415 event that it discovers misbehavior from a log. Misbehavior from 1416 a log involves the log being unable to provide either (a) a 1417 consistency proof between two valid STHs or (b) an inclusion proof 1418 for a certificate to an STH any time after the log's MMD has 1419 elapsed from the issuance of the SCT. The log's inability to 1420 provide either proof will not be externally cryptographically- 1421 verifiable, as it may be indistinguishable from a network error. 1423 11. Policy Recommendations 1425 This section is intended as suggestions to implementors of HTTPS 1426 Clients, HTTPS servers, and CT auditors. It is not a requirement for 1427 technique of implementation, so long as the privacy considerations 1428 established above are obeyed. 1430 11.1. Blocking Recommendations 1432 11.1.1. Frustrating blocking 1434 When making gossip connections to HTTPS servers or Trusted Auditors, 1435 it is desirable to minimize the plaintext metadata in the connection 1436 that can be used to identify the connection as a gossip connection 1437 and therefore be of interest to block. Additionally, introducing 1438 some randomness into client behavior may be important. We assume 1439 that the adversary is able to inspect the behavior of the HTTPS 1440 client and understand how it makes gossip connections. 1442 As an example, if a client, after establishing a TLS connection (and 1443 receiving an SCT, but not making its own HTTP request yet), 1444 immediately opens a second TLS connection for the purpose of gossip, 1445 the adversary can reliably block this second connection to block 1446 gossip without affecting normal browsing. For this reason it is 1447 recommended to run the gossip protocols over an existing connection 1448 to the server, making use of connection multiplexing such as HTTP 1449 Keep-Alive or SPDY. 1451 Truncation is also a concern. If a client always establishes a TLS 1452 connection, makes a request, receives a response, and then always 1453 attempts a gossip communication immediately following the first 1454 response, truncation will allow an attacker to block gossip reliably. 1456 For these reasons, we recommend that, if at all possible, clients 1457 SHOULD send gossip data in an already established TLS session. This 1458 can be done through the use of HTTP Pipelining, SPDY, or HTTP/2. 1460 11.1.2. Responding to possible blocking 1462 In some circumstances a client may have a piece of data that they 1463 have attempted to share (via SCT Feedback or STH Pollination), but 1464 have been unable to do so: with every attempt they receive an error. 1465 These situations are: 1467 1. The client has an SCT and a certificate, and attempts to retrieve 1468 an inclusion proof - but receives an error on every attempt. 1470 2. The client has an STH, and attempts to resolve it to a newer STH 1471 via a consistency proof - but receives an error on every attempt. 1473 3. The client has attempted to share an SCT and constructed 1474 certificate via SCT Feedback - but receives an error on every 1475 attempt. 1477 4. The client has attempted to share an STH via STH Pollination - 1478 but receives an error on every attempt. 1480 5. The client has attempted to share a specific piece of data with a 1481 Trusted Auditor - but receives an error on every attempt. 1483 In the case of 1 or 2, it is conceivable that the reason for the 1484 errors is that the log acted improperly, either through malicious 1485 actions or compromise. A proof may not be able to be fetched because 1486 it does not exist (and only errors or timeouts occur). One such 1487 situation may arise because of an actively malicious log, as 1488 presented in Section 10.1. This data is especially important to 1489 share with the broader internet to detect this situation. 1491 If an SCT has attempted to be resolved to an STH via an inclusion 1492 proof multiple times, and each time has failed, this SCT might very 1493 well be a compromising proof of an attack. However the client MUST 1494 NOT share the data with any other third party (excepting a Trusted 1495 Auditor should one exist). 1497 If an STH has attempted to be resolved to a newer STH via a 1498 consistency proof multiple times, and each time has failed, a client 1499 MAY share the STH with an "Auditor of Last Resort" even if the STH in 1500 question is no longer within the validity window. This auditor may 1501 be pre-configured in the client, but the client SHOULD permit a user 1502 to disable the functionality or change whom data is sent to. The 1503 Auditor of Last Resort itself represents a point of failure and 1504 privacy concerns, so if implemented, it SHOULD connect using public 1505 key pinning and not consider an item delivered until it receives a 1506 confirmation. 1508 In the cases 3, 4, and 5, we assume that the webserver(s) or trusted 1509 auditor in question is either experiencing an operational failure, or 1510 being attacked. In both cases, a client SHOULD retain the data for 1511 later submission (subject to Private Browsing or other history- 1512 clearing actions taken by the user.) This is elaborated upon more in 1513 Section 11.3. 1515 11.2. Proof Fetching Recommendations 1517 Proof fetching (both inclusion proofs and consistency proofs) SHOULD 1518 be performed at random time intervals. If proof fetching occurred 1519 all at once, in a flurry of activity, a log would know that SCTs or 1520 STHs received around the same time are more likely to come from a 1521 particular client. While proof fetching is required to be done in a 1522 manner that attempts to be anonymous from the perspective of the log, 1523 the correlation of activity to a single client would still reveal 1524 patterns of user behavior we wish to keep confidential. These 1525 patterns could be recognizable as a single user, or could reveal what 1526 sites are commonly visited together in the aggregate. 1528 11.3. Record Distribution Recommendations 1530 In several components of the CT Gossip ecosystem, the recommendation 1531 is made that data from multiple sources be ingested, mixed, stored 1532 for an indeterminate period of time, provided (multiple times) to a 1533 third party, and eventually deleted. The instances of these 1534 recommendations in this draft are: 1536 o When a client receives SCTs during SCT Feedback, it should store 1537 the SCTs and Certificate Chain for some amount of time, provide 1538 some of them back to the server at some point, and may eventually 1539 remove them from its store 1541 o When a client receives STHs during STH Pollination, it should 1542 store them for some amount of time, mix them with other STHs, 1543 release some of them them to various servers at some point, 1544 resolve some of them to new STHs, and eventually remove them from 1545 its store 1547 o When a server receives SCTs during SCT Feedback, it should store 1548 them for some period of time, provide them to auditors some number 1549 of times, and may eventually remove them 1551 o When a server receives STHs during STH Pollination, it should 1552 store them for some period of time, mix them with other STHs, 1553 provide some of them to connecting clients, may resolve them to 1554 new STHs via Proof Fetching, and eventually remove them from its 1555 store 1557 o When a Trusted Auditor receives SCTs or historical STHs from 1558 clients, it should store them for some period of time, mix them 1559 with SCTs received from other clients, and act upon them at some 1560 period of time 1562 Each of these instances have specific requirements for user privacy, 1563 and each have options that may not be invoked. As one example, an 1564 HTTPS client should not mix SCTs from server A with SCTs from server 1565 B and release server B's SCTs to Server A. As another example, an 1566 HTTPS server may choose to resolve STHs to a single more current STH 1567 via proof fetching, but it is under no obligation to do so. 1569 These requirements should be met, but the general problem of 1570 aggregating multiple pieces of data, choosing when and how many to 1571 release, and when to remove them is shared. This problem has 1572 previously been considered in the case of Mix Networks and Remailers, 1573 including papers such as [trickle]. 1575 There are several concerns to be addressed in this area, outlined 1576 below. 1578 11.3.1. Mixing Algorithm 1580 When SCTs or STHs are recorded by a participant in CT Gossip and 1581 later used, it is important that they are selected from the datastore 1582 in a non-deterministic fashion. 1584 This is most important for servers, as they can be queried for SCTs 1585 and STHs anonymously. If the server used a predictable ordering 1586 algorithm, an attacker could exploit the predictability to learn 1587 information about a client. One such method would be by observing 1588 the (encrypted) traffic to a server. When a client of interest 1589 connects, the attacker makes a note. They observe more clients 1590 connecting, and predicts at what point the client-of-interest's data 1591 will be disclosed, and ensures that they query the server at that 1592 point. 1594 Although most important for servers, random ordering is still 1595 strongly recommended for clients and Trusted Auditors. The above 1596 attack can still occur for these entities, although the circumstances 1597 are less straightforward. For clients, an attacker could observe 1598 their behavior, note when they receive an STH from a server, and use 1599 javascript to cause a network connection at the correct time to force 1600 a client to disclose the specific STH. Trusted Auditors are stewards 1601 of sensitive client data. If an attacker had the ability to observe 1602 the activities of a Trusted Auditor (perhaps by being a log, or 1603 another auditor), they could perform the same attack - noting the 1604 disclosure of data from a client to the Trusted Auditor, and then 1605 correlating a later disclosure from the Trusted Auditor as coming 1606 from that client. 1608 Random ordering can be ensured by several mechanisms. A datastore 1609 can be shuffled, using a secure shuffling algorithm such as Fisher- 1610 Yates. Alternately, a series of random indexes into the data store 1611 can be selected (if a collision occurs, a new index is selected.) A 1612 cryptographically secure random number generator must be used in 1613 either case. If shuffling is performed, the datastore must be marked 1614 'dirty' upon item insertion, and at least one shuffle operation 1615 occurs on a dirty datastore before data is retrieved from it for use. 1617 11.3.2. The Deletion Algorithm 1619 No entity in CT Gossip is required to delete records at any time, 1620 except to respect user's wishes such as private browsing mode or 1621 clearing history. However, it is likely that over time the 1622 accumulated storage will grow in size and need to be pruned. 1624 While deletion of data will occur, proof fetching can ensure that any 1625 misbehavior from a log will still be detected, even after the direct 1626 evidence from the attack is deleted. Proof fetching ensures that if 1627 a log presents a split view for a client, they must maintain that 1628 split view in perpetuity. An inclusion proof from an SCT to an STH 1629 does not erase the evidence - the new STH is evidence itself. A 1630 consistency proof from that STH to a new one likewise - the new STH 1631 is every bit as incriminating as the first. (Client behavior in the 1632 situation where an SCT or STH cannot be resolved is suggested in 1633 Section 11.1.2.) Because of this property, we recommend that if a 1634 client is performing proof fetching, that they make every effort to 1635 not delete data until it has been successfully resolved to a new STH 1636 via a proof. 1638 When it is time to delete a record, it can be done in a way that 1639 makes it more difficult for a successful flushing attack to to be 1640 performed. 1642 1. When the record cache has reached a certain size that is yet 1643 under the limit, aggressively perform proof fetching. This 1644 should resolve records to a small set of STHs that can be 1645 retained. Once a proof has been fetched, the record is safer to 1646 delete. 1648 2. If proof fetching has failed, or is disabled, begin by deleting 1649 SCTs and Certificate Chains that have been successfully reported. 1650 Deletion from this set of SCTs should be done at random. For a 1651 client, a submission is not counted as being reported unless it 1652 is sent over a connection using a different SCT, so the attacker 1653 is faced with a recursive problem. (For a server, this step does 1654 not apply.) 1656 3. Attempt to save any submissions that have failed proof fetching 1657 repeatedly, as these are the most likely to be indicative of an 1658 attack. 1660 4. Finally, if the above steps have been followed and have not 1661 succeeded in reducing the size sufficiently, records may be 1662 deleted at random. 1664 Note that if proof fetching is disabled (which is expected although 1665 not required for servers) - the algorithm collapses down to 'delete 1666 at random'. 1668 The decision to delete records at random is intentional. Introducing 1669 non-determinism in the decision is absolutely necessary to make it 1670 more difficult for an adversary to know with certainty or high 1671 confidence that the record has been successfully flushed from a 1672 target. 1674 11.4. Concrete Recommendations 1676 We present the following pseudocode as a concrete outline of our 1677 policy recommendations. 1679 Both suggestions presented are applicable to both clients and 1680 servers. Servers may not perform proof fetching, in which case large 1681 portions of the pseudocode are not applicable. But it should work in 1682 either case. 1684 Note that we use a function 'rand()' in the pseudocode, this function 1685 is assumed to be a cryptographically secure pseudorandom number 1686 generator. Additionally, when N unique items are needed, they are 1687 chosen at random by drawing a random index repeatedly until the N 1688 unique items from an array have been chosen. Although simple, when 1689 the array is N or near-N items in length this is inefficient. A 1690 secure shuffle algorithm followed by selecting the first N items may 1691 be more efficient, especially when N is large. 1693 11.4.1. STH Pollination 1695 The STH class contains data pertaining specifically to the STH 1696 itself. 1698 class STH 1699 { 1700 uint16 proof_attempts 1701 uint16 proof_failure_count 1702 uint32 num_reports_to_thirdparty 1703 datetime timestamp 1704 byte[] data 1705 } 1707 The broader STH store itself would contain all the STHs known by an 1708 entity participating in STH Pollination (either client or server). 1709 This simplistic view of the class does not take into account the 1710 complicated locking that would likely be required for a data 1711 structure being accessed by multiple threads. Something to note 1712 about this pseudocode is that it does not remove STHs once they have 1713 been resolved to a newer STH. Doing so might make older STHs within 1714 the validity window rarer and thus enable tracking. 1716 class STHStore 1717 { 1718 STH[] sth_list 1720 // This function is run after receiving a set of STHs from 1721 // a third party in response to a pollination submission 1722 def insert(STH[] new_sths) { 1723 foreach(new in new_sths) { 1724 if(this.sth_list.contains(new)) 1725 continue 1726 this.sth_list.insert(new) 1727 } 1728 } 1730 // This function is called to delete the given STH 1731 // from the data store 1732 def delete_now(STH s) { 1733 this.sth_list.remove(s) 1734 } 1736 // When it is time to perform STH Pollination, the HTTPS client 1737 // calls this function to get a selection of STHs to send as 1738 // feedback 1739 def get_pollination_selection() { 1740 if(len(this.sth_list) < MAX_STH_TO_GOSSIP) 1741 return this.sth_list 1742 else { 1743 indexes = set() 1744 modulus = len(this.sth_list) 1745 outdated_sths = 0 1746 while(len(indexes) + outdated_sths < MAX_STH_TO_GOSSIP) { 1747 r = randomInt() % modulus 1748 if(r not in indexes) 1749 // Ignore STHs that are past the validity window but not 1750 // yet removed. 1751 if(now() - this.sth_list[i].timestamp < TWO_WEEKS) 1752 outdated_sths++; 1753 else 1754 indexes.insert(r) 1755 } 1757 return_selection = [] 1758 foreach(i in indexes) { 1759 return_selection.insert(this.sth_list[i]) 1760 } 1761 return return_selection 1762 } 1763 } 1764 } 1766 We also suggest a function that will be called periodically in the 1767 background, iterating through the STH store, performing a cleaning 1768 operation and queuing consistency proofs. This function can live as 1769 a member functions of the STHStore class. 1771 //Just a suggestion: 1772 #define MIN_PROOF_FAILURES_CONSIDERED_SUSPICIOUS 3 1774 def clean_list() { 1775 foreach(sth in this.sth_list) { 1777 if(now() - sth.timestamp > TWO_WEEKS) { 1778 //STH is too old, we must remove it 1779 if(proof_fetching_enabled 1780 && auditor_of_last_resort_enabled 1781 && sth.proof_failure_count 1782 > MIN_PROOF_FAILURES_CONSIDERED_SUSPICIOUS) { 1783 queue_for_auditor_of_last_resort(sth, 1784 auditor_of_last_resort_callback) 1785 } else { 1786 delete_now(sth) 1787 } 1788 } 1790 else if(proof_fetching_enabled 1791 && now() - sth.timestamp > LOG_MMD 1792 && sth.proof_attempts != UINT16_MAX 1793 // Only fetch a proof is we have never received a proof 1794 // before. (This also avoids submitting something 1795 // already in the queue.) 1796 && sth.proof_attempts == sth.proof_failure_count) { 1797 sth.proof_attempts++ 1798 queue_consistency_proof(sth, consistency_proof_callback) 1799 } 1800 } 1801 } 1803 These functions also exist in the STHStore class. 1805 // This function is called after successfully pollinating STHs 1806 // to a third party. It is passed the STHs sent to the third 1807 // party, which is the output of get_gossip_selection(), as well 1808 // as the STHs received in the response. 1809 def successful_thirdparty_submission_callback(STH[] submitted_sth_list, 1810 STH[] new_sths) 1811 { 1812 foreach(sth in submitted_sth_list) { 1813 sth.num_reports_to_thirdparty++ 1814 } 1816 this.insert(new_sths); 1817 } 1819 // Attempt auditor of last resort submissions until it succeeds 1820 def auditor_of_last_resort_callback(original_sth, error) { 1821 if(!error) { 1822 delete_now(original_sth) 1823 } 1824 } 1826 def consistency_proof_callback(consistency_proof, original_sth, error) { 1827 if(!error) { 1828 insert(consistency_proof.current_sth) 1829 } else { 1830 original_sth.proof_failure_count++ 1831 } 1832 } 1834 11.4.2. SCT Feedback 1836 The SCT class contains data pertaining specifically to an SCT itself. 1838 class SCT 1839 { 1840 uint16 proof_failure_count 1841 bool has_been_resolved_to_sth 1842 bool proof_outstanding 1843 byte[] data 1844 } 1846 The SCT bundle will contain the trusted certificate chain the HTTPS 1847 client built (chaining to a trusted root certificate.) It also 1848 contains the list of associated SCTs, the exact domain it is 1849 applicable to, and metadata pertaining to how often it has been 1850 reported to the third party. 1852 class SCTBundle 1853 { 1854 X509[] certificate_chain 1855 SCT[] sct_list 1856 string domain 1857 uint32 num_reports_to_thirdparty 1859 def equals(sct_bundle) { 1860 if(sct_bundle.domain != this.domain) 1861 return false 1862 if(sct_bundle.certificate_chain != this.certificate_chain) 1863 return false 1864 if(sct_bundle.sct_list != this.sct_list) 1865 return false 1867 return true 1868 } 1869 def approx_equals(sct_bundle) { 1870 if(sct_bundle.domain != this.domain) 1871 return false 1872 if(sct_bundle.certificate_chain != this.certificate_chain) 1873 return false 1875 return true 1876 } 1878 def insert_scts(sct[] sct_list) { 1879 this.sct_list.union(sct_list) 1880 this.num_reports_to_thirdparty = 0 1881 } 1883 def has_been_fully_resolved_to_sths() { 1884 foreach(s in this.sct_list) { 1885 if(!s.has_been_resolved_to_sth && !s.proof_outstanding) 1886 return false 1887 } 1888 return true 1889 } 1891 def max_proof_failures() { 1892 uint max = 0 1893 foreach(sct in this.sct_list) { 1894 if(sct.proof_failure_count > max) 1895 max = sct.proof_failure_count 1896 } 1897 return max 1898 } 1899 } 1900 For each domain, we store a SCTDomainEntry that holds the SCTBundles 1901 seen for that domain, as well as encapsulating some logic relating to 1902 SCT Feedback for that particular domain. In particular, this data 1903 structure also contains the logic that handles domains not supporting 1904 SCT Feedback. Its behavior is: 1906 1. When a user visits a domain, SCT Feedback is attempted for it. 1907 If it fails, it will retry after a month (configurable). If it 1908 succeeds, excellent. SCT Feedback data is still collected and 1909 stored even if SCT Feedback failed. 1911 2. After 3 month-long waits between failures, the domain will be 1912 marked as failing long-term. No SCT Feedback data will be stored 1913 beyond meta-data, but SCT Feedback will still be attempted after 1914 month-long waits 1916 3. If at any point in time, SCT Feedback succeeds, all failure 1917 counters are reset 1919 4. If a domain succeeds, but then begins failing, it must fail more 1920 than 90% of the time (configurable) and then the process begins 1921 at (2). 1923 If a domain is visited infrequently (say, once every 7 months) then 1924 it will be evicted from the cache and start all over again (according 1925 to the suggestion values in the below pseudocode). 1927 //Suggestions: 1928 // After concluding a domain doesn't support feedback, we try again 1929 // after WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS amount of time to see if 1930 // they added support 1931 #define WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS 1 month 1933 // If we've waited MIN_SCT_FEEDBACK_ATTEMPTS_BEFORE_OMITTING_STORAGE 1934 // multiplied by WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS amount of time, we 1935 // still attempt SCT Feedback, but no longer bother storing any data 1936 // until the domain supports SCT Feedback 1937 #define MIN_SCT_FEEDBACK_ATTEMPTS_BEFORE_OMITTING_STORAGE 3 1939 // If this percentage of SCT Feedback attempts previously succeeded, 1940 // we consider the domain as supporting feedback and is just having 1941 // transient errors 1942 #define MIN_RATIO_FOR_SCT_FEEDBACK_TO_BE_WORKING .10 1944 class SCTDomainEntry 1945 { 1946 // This is the primary key of the object, the exact domain name it 1947 // is valid for 1948 string domain 1950 // This is the last time the domain was contacted. For client 1951 // operations it is updated whenever the client makes any request 1952 // (not just feedback) to the domain. For server operations, it is 1953 // updated whenever any client contacts the domain. Responsibility 1954 // for updating lies OUTSIDE of the class 1955 public datetime last_contact_for_domain 1957 // This is the last time SCT Feedback was attempted for the domain. 1958 // It is updated whenever feedback is attempted - responsibility for 1959 // updating lies OUTSIDE of the class 1960 // This is not used when this algorithm runs on servers 1961 public datetime last_sct_feedback_attempt 1963 // This is the number of times we have waited an 1964 // WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS amount of time, and still failed 1965 // e.g., 10 months of failures 1966 // This is not used when this algorithm runs on servers 1967 private uint16 num_feedback_loop_failures 1969 // This is whether or not SCT Feedback has failed enough times that we 1970 // should not bother storing data for it anymore. It is a small 1971 // function used for illustrative purposes. 1972 // This is not used when this algorithm runs on servers 1973 private bool sct_feedback_failing_longterm() 1974 { num_feedback_loop_failures >= 1975 MIN_SCT_FEEDBACK_ATTEMPTS_BEFORE_OMITTING_STORAGE } 1977 // This is the number of SCT Feedback submissions attempted. 1978 // Responsibility for incrementing lies OUTSIDE of the class 1979 // (And watch for integer overflows) 1980 // This is not used when this algorithm runs on servers 1981 public uint16 num_submissions_attempted 1983 // This is the number of successful SCT Feedback submissions. This 1984 // variable is updated by the class. 1985 // This is not used when this algorithm runs on servers 1986 private uint16 num_submissions_succeeded 1988 // This contains all the bundles of SCT data we have observed for 1989 // this domain 1990 SCTBundle[] observed_records 1992 // This function can be called to determine if we should attempt 1993 // SCT Feedback for this domain. 1994 def should_attempt_feedback() { 1995 // Servers always perform feedback! 1996 if(operator_is_server) 1997 return true 1999 // If we have not tried in a month, try again 2000 if(now() - last_sct_feedback_attempt > 2001 WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS) 2002 return true 2004 // If we have tried recently, and it seems to be working, go for it! 2005 if((num_submissions_succeeded / num_submissions_attempted) > 2006 MIN_RATIO_FOR_SCT_FEEDBACK_TO_BE_WORKING) 2007 return true 2009 // Otherwise don't try 2010 return false 2011 } 2013 // For Clients, this function is called after a successful 2014 // connection to an HTTPS server, with a single SCTBundle 2015 // constructed from that connection's certificate chain and SCTs. 2016 // For Servers, this is called after receiving SCT Feedback with 2017 // all the bundles sent in the feedback. 2018 def insert(SCTBundle[] bundles) { 2019 // Do not store data for long-failing domains 2020 if(sct_feedback_failing_longterm()) { 2021 return 2022 } 2024 foreach(b in bundles) { 2025 if(operator_is_server) { 2026 if(!passes_validity_checks(b)) 2027 return 2028 } 2030 bool have_inserted = false 2031 foreach(e in this.observed_records) { 2032 if(e.equals(b)) 2033 return 2034 else if(e.approx_equals(b)) { 2035 have_inserted = true 2036 e.insert_scts(b.sct_list) 2037 } 2038 } 2039 if(!have_inserted) 2040 this.observed_records.insert(b) 2041 } 2042 SCTStoreManager.update_cache_percentage() 2044 } 2046 // When it is time to perform SCT Feedback, the HTTPS client 2047 // calls this function to get a selection of SCTBundles to send 2048 // as feedback 2049 def get_gossip_selection() { 2050 if(len(observed_records) > MAX_SCT_RECORDS_TO_GOSSIP) { 2051 indexes = set() 2052 modulus = len(observed_records) 2053 while(len(indexes) < MAX_SCT_RECORDS_TO_GOSSIP) { 2054 r = randomInt() % modulus 2055 if(r not in indexes) 2056 indexes.insert(r) 2057 } 2059 return_selection = [] 2060 foreach(i in indexes) { 2061 return_selection.insert(this.observed_records[i]) 2062 } 2064 return return_selection 2065 } 2066 else 2067 return this.observed_records 2068 } 2070 def passes_validity_checks(SCTBundle b) { 2071 // This function performs the validity checks specified in 2072 // {{feedback-srvop}} 2073 } 2074 } 2076 The SCTDomainEntry is responsible for handling the outcome of a 2077 submission report for that domain using its member function: 2079 // This function is called after providing SCT Feedback 2080 // to a server. It is passed the feedback sent to the other party, which 2081 // is the output of get_gossip_selection(), and also the SCTBundle 2082 // representing the connection the data was sent on. 2083 // (When this code runs on the server, connectionBundle is NULL) 2084 // If the Feedback was not sent successfully, error is True 2085 def after_submit_to_thirdparty(error, SCTBundle[] submittedBundles, 2086 SCTBundle connectionBundle) 2087 { 2088 // Server operation in this instance is exceedingly simple 2089 if(operator_is_server) { 2090 if(error) 2091 return 2093 foreach(bundle in submittedBundles) 2094 bundle.num_reports_to_thirdparty++ 2095 return 2096 } 2098 // Client behavior is much more complicated 2099 if(error) { 2100 if(sct_feedback_failing_longterm()) { 2101 num_feedback_loop_failures++ 2102 } 2103 else if((num_submissions_succeeded / num_submissions_attempted) 2104 > MIN_RATIO_FOR_SCT_FEEDBACK_TO_BE_WORKING) { 2105 // Do nothing. num_submissions_succeeded will not be incremented 2106 // After enough of these failures, the ratio will fall beyond 2107 // acceptable 2108 } else { 2109 // The domain has begun its three-month grace period. We will 2110 // attempt submissions once a month 2111 num_feedback_loop_failures++ 2112 } 2113 return 2114 } 2115 // We succeeded, so reset all of our failure states 2116 // Note, there is a race condition here if clear_old_data() is called 2117 // while this callback is outstanding. 2118 num_feedback_loop_failures = 0 2119 if(num_submissions_succeeded != UINT16_MAX ) 2120 num_submissions_succeeded++ 2122 foreach(bundle in submittedBundles) 2123 { 2124 // Compare Certificate Chains, if they do not match, it counts as a 2125 // submission. 2126 if(!connectionBundle.approx_equals(bundle)) 2127 bundle.num_reports_to_thirdparty++ 2128 else { 2129 // This check ensures that a SCT Bundle is not considered reported 2130 // if it is submitted over a connection with the same SCTs. This 2131 // satisfies the constraint in Paragraph 5 of {{feedback-clisrv}} 2132 // Consider three submission scenarios: 2133 // Submitted SCTs Connection SCTs Considered Submitted 2134 // A, B A, B No - no new information 2135 // A A, B Yes - B is a new SCT 2136 // A, B A No - no new information 2137 if(connectionBundle.sct_list is NOT a subset of bundle.sct_list) 2138 bundle.num_reports_to_thirdparty++ 2139 } 2141 } 2142 } 2144 Instances of the SCTDomainEntry class are stored as part of a larger 2145 class that manages the entire SCT Cache, storing them in a hashmap 2146 keyed by domain. This class also tracks the current size of the 2147 cache, and will trigger cache eviction. 2149 //Suggestions: 2150 #define CACHE_PRESSURE_SAFE .50 2151 #define CACHE_PRESSURE_IMMINENT .70 2152 #define CACHE_PRESSURE_ALMOST_FULL .85 2153 #define CACHE_PRESSURE_FULL .95 2154 #define WAIT_BETWEEN_IMMINENT_CACHE_EVICTION 5 minutes 2156 class SCTStoreManager 2157 { 2158 hashmap all_sct_entries 2159 uint32 current_cache_size 2160 datetime imminent_cache_pressure_check_performed 2162 float current_cache_percentage() { 2163 return current_cache_size / MAX_CACHE_SIZE; 2164 } 2166 static def update_cache_percentage() { 2167 // This function calculates the current size of the cache 2168 // and updates current_cache_size 2169 /* ... perform calculations ... */ 2170 current_cache_size = /* new calculated value */ 2172 // Perform locking to prevent multiple of these functions being 2173 // called concurrently or unnecessarily 2174 if(current_cache_percentage() > CACHE_PRESSURE_FULL) { 2175 cache_is_full() 2176 } 2178 else if(current_cache_percentage() > CACHE_PRESSURE_ALMOST_FULL) { 2179 cache_pressure_almost_full() 2180 } 2182 else if(current_cache_percentage() > CACHE_PRESSURE_IMMINENT) { 2183 // Do not repeatedly perform the imminent cache pressure operation 2184 if(now() - imminent_cache_pressure_check_performed > 2185 WAIT_BETWEEN_IMMINENT_CACHE_EVICTION) { 2186 cache_pressure_is_imminent() 2187 } 2188 } 2189 } 2190 } 2192 The SCTStoreManager contains a function that will be called 2193 periodically in the background, iterating through all SCTDomainEntry 2194 objects and performing maintenance tasks. It removes data for 2195 domains we have not contacted in a long time. This function is not 2196 intended to clear data if the cache is getting full, separate 2197 functions are used for that. 2199 // Suggestions: 2200 #define TIME_UNTIL_OLD_SUBMITTED_SCTDATA_ERASED 3 months 2201 #define TIME_UNTIL_OLD_UNSUBMITTED_SCTDATA_ERASED 6 months 2203 def clear_old_data() 2204 { 2205 foreach(domainEntry in all_sct_stores) 2206 { 2207 // Queue proof fetches 2208 if(proof_fetching_enabled) { 2209 foreach(sctBundle in domainEntry.observed_records) { 2210 if(!sctBundle.has_been_fully_resolved_to_sths()) { 2211 foreach(s in bundle.sct_list) { 2212 if(!s.has_been_resolved_to_sth && !s.proof_outstanding) { 2213 sct.proof_outstanding = True 2214 queue_inclusion_proof(sct, inclusion_proof_callback) 2215 } 2216 } 2217 } 2218 } 2219 } 2221 // Do not store data for domains who are not supporting SCT 2222 if(!operator_is_server 2223 && domainEntry.sct_feedback_failing_longterm()) 2224 { 2225 // Note that reseting these variables every single time is 2226 // necessary to avoid a bug 2227 all_sct_stores[domainEntry].num_submissions_attempted = 0 2228 all_sct_stores[domainEntry].num_submissions_succeeded = 0 2229 delete all_sct_stores[domainEntry].observed_records 2230 all_sct_stores[domainEntry].observed_records = NULL 2231 } 2233 // This check removes successfully submitted data for 2234 // old domains we have not dealt with in a long time 2235 if(domainEntry.num_submissions_succeeded > 0 2236 && now() - domainEntry.last_contact_for_domain 2237 > TIME_UNTIL_OLD_SUBMITTED_SCTDATA_ERASED) 2238 { 2239 all_sct_stores.remove(domainEntry) 2240 } 2242 // This check removes unsuccessfully submitted data for 2243 // old domains we have not dealt with in a very long time 2244 if(now() - domainEntry.last_contact_for_domain 2245 > TIME_UNTIL_OLD_UNSUBMITTED_SCTDATA_ERASED) 2246 { 2247 all_sct_stores.remove(domainEntry) 2248 } 2250 SCTStoreManager.update_cache_percentage() 2251 } 2253 Inclusion Proof Fetching is handled fairly independently 2255 // This function is a callback invoked after an inclusion proof 2256 // has been retrieved. It can exist on the SCT class or independently, 2257 // so long as it can modify the SCT class' members 2258 def inclusion_proof_callback(inclusion_proof, original_sct, error) 2259 { 2260 // Unlike the STH code, this counter must be incremented on the 2261 // callback as there is a race condition on using this counter in the 2262 // cache_* functions. 2263 original_sct.proof_attempts++ 2264 original_sct.proof_outstanding = False 2265 if(!error) { 2266 original_sct.has_been_resolved_to_sth = True 2267 insert_to_sth_datastore(inclusion_proof.new_sth) 2268 } else { 2269 original_sct.proof_failure_count++ 2270 } 2271 } 2273 If the cache is getting full, these three member functions of the 2274 SCTStoreManager class will be used. 2276 // ----------------------------------------------------------------- 2277 // This function is called when the cache is not yet full, but is 2278 // nearing it. It prioritizes deleting data that should be safe 2279 // to delete (because it has been shared with the site or resolved 2280 // to an STH) 2281 def cache_pressure_is_imminent() 2282 { 2283 bundlesToDelete = [] 2284 foreach(domainEntry in all_sct_stores) { 2285 foreach(sctBundle in domainEntry.observed_records) { 2287 if(proof_fetching_enabled) { 2288 // First, queue proofs for anything not already queued. 2289 if(!sctBundle.has_been_fully_resolved_to_sths()) { 2290 foreach(sct in bundle.sct_list) { 2291 if(!sct.has_been_resolved_to_sth 2292 && !sct.proof_outstanding) { 2293 sct.proof_outstanding = True 2294 queue_inclusion_proof(sct, inclusion_proof_callback) 2295 } 2296 } 2297 } 2299 // Second, consider deleting entries that have been fully 2300 // resolved. 2301 else { 2302 bundlesToDelete.append( Struct(domainEntry, sctBundle) ) 2303 } 2304 } 2306 // Third, consider deleting entries that have been successfully 2307 // reported 2308 if(sctBundle.num_reports_to_thirdparty > 0) { 2309 bundlesToDelete.append( Struct(domainEntry, sctBundle) ) 2310 } 2311 } 2312 } 2314 // Third, delete the eligible entries at random until the cache is 2315 // at a safe level 2316 uint recalculateIndex = 0 2317 #define RECALCULATE_EVERY_N_OPERATIONS 50 2319 while(bundlesToDelete.length > 0 && 2320 current_cache_percentage() > CACHE_PRESSURE_SAFE) { 2321 uint rndIndex = rand() % bundlesToDelete.length 2322 bundlesToDelete[rndIndex].domainEntry.observed_records.remove( 2323 bundlesToDelete[rndIndex].sctBundle) 2324 bundlesToDelete.removeAt(rndIndex) 2326 recalculateIndex++ 2327 if(recalculateIndex % RECALCULATE_EVERY_N_OPERATIONS == 0) { 2328 update_cache_percentage() 2329 } 2330 } 2332 // Finally, tell the proof fetching engine to go faster 2333 if(proof_fetching_enabled) { 2334 // This function would speed up proof fetching until an 2335 // arbitrary time has passed. Perhaps until it has fetched 2336 // proofs for the number of items currently in its queue? Or 2337 // a percentage of them? 2338 proof_fetch_faster_please() 2339 } 2340 update_cache_percentage(); 2341 } 2343 // ----------------------------------------------------------------- 2344 // This function is called when the cache is almost full. It will 2345 // evict entries at random, while attempting to save entries that 2346 // appear to have proof fetching failures 2347 def cache_pressure_almost_full() 2348 { 2349 uint recalculateIndex = 0 2350 uint savedRecords = 0 2351 #define RECALCULATE_EVERY_N_OPERATIONS 50 2353 while(all_sct_stores.length > savedRecords && 2354 current_cache_percentage() > CACHE_PRESSURE_SAFE) { 2355 uint rndIndex1 = rand() % all_sct_stores.length 2356 uint rndIndex2 = rand() % 2357 all_sct_stores[rndIndex1].observed_records.length 2359 if(proof_fetching_enabled) { 2360 if(all_sct_stores[rndIndex1].observed_records[ 2361 rndIndex2].max_proof_failures() > 2362 MIN_PROOF_FAILURES_CONSIDERED_SUSPICIOUS) { 2363 savedRecords++ 2364 continue 2365 } 2366 } 2368 // If proof fetching is not enabled we need some other logic 2369 else { 2370 if(sctBundle.num_reports_to_thirdparty == 0) { 2371 savedRecords++ 2372 continue 2373 } 2374 } 2376 all_sct_stores[rndIndex1].observed_records.removeAt(rndIndex2) 2377 if(all_sct_stores[rndIndex1].observed_records.length == 0) { 2378 all_sct_stores.removeAt(rndIndex1) 2379 } 2381 recalculateIndex++ 2382 if(recalculateIndex % RECALCULATE_EVERY_N_OPERATIONS == 0) { 2383 update_cache_percentage() 2384 } 2385 } 2387 update_cache_percentage(); 2389 } 2391 // ----------------------------------------------------------------- 2392 // This function is called when the cache is full, and will evict 2393 // cache entries at random 2394 def cache_is_full() 2395 { 2396 uint recalculateIndex = 0 2397 #define RECALCULATE_EVERY_N_OPERATIONS 50 2399 while(all_sct_stores.length > 0 && 2400 current_cache_percentage() > CACHE_PRESSURE_SAFE) { 2401 uint rndIndex1 = rand() % all_sct_stores.length 2402 uint rndIndex2 = rand() % 2403 all_sct_stores[rndIndex1].observed_records.length 2405 all_sct_stores[rndIndex1].observed_records.removeAt(rndIndex2) 2406 if(all_sct_stores[rndIndex1].observed_records.length == 0) { 2407 all_sct_stores.removeAt(rndIndex1) 2408 } 2410 recalculateIndex++ 2411 if(recalculateIndex % RECALCULATE_EVERY_N_OPERATIONS == 0) { 2412 update_cache_percentage() 2413 } 2414 } 2416 update_cache_percentage(); 2417 } 2419 12. IANA considerations 2421 There are no IANA considerations. 2423 13. Contributors 2425 The authors would like to thank the following contributors for 2426 valuable suggestions: Al Cutter, Andrew Ayer, Ben Laurie, Benjamin 2427 Kaduk, Graham Edgecombe, Josef Gustafsson, Karen Seo, Magnus Ahltorp, 2428 Steven Kent, Yan Zhu. 2430 14. ChangeLog 2431 14.1. Changes between ietf-04 and ietf-05 2433 o STH and SCT data formats changed to support CT v1 and v2. 2435 o Address ED review comments. 2437 14.2. Changes between ietf-03 and ietf-04 2439 o No changes. 2441 14.3. Changes between ietf-02 and ietf-03 2443 o TBD's resolved. 2445 o References added. 2447 o Pseduocode changed to work for both clients and servers. 2449 14.4. Changes between ietf-01 and ietf-02 2451 o Requiring full certificate chain in SCT Feedback. 2453 o Clarifications on what clients store for and send in SCT Feedback 2454 added. 2456 o SCT Feedback server operation updated to protect against DoS 2457 attacks on servers. 2459 o Pre-Loaded vs Locally Added Anchors explained. 2461 o Base for well-known URL's changed. 2463 o Remove all mentions of monitors - gossip deals with auditors. 2465 o New sections added: Trusted Auditor protocol, attacks by actively 2466 malicious log, the Dual-CA compromise attack, policy 2467 recommendations, 2469 14.5. Changes between ietf-00 and ietf-01 2471 o Improve language and readability based on feedback from Stephen 2472 Kent. 2474 o STH Pollination Proof Fetching defined and indicated as optional. 2476 o 3-Method Ecosystem section added. 2478 o Cases with Logs ceasing operation handled. 2480 o Text on tracking via STH Interaction added. 2482 o Section with some early recommendations for mixing added. 2484 o Section detailing blocking connections, frustrating it, and the 2485 implications added. 2487 14.6. Changes between -01 and -02 2489 o STH Pollination defined. 2491 o Trusted Auditor Relationship defined. 2493 o Overview section rewritten. 2495 o Data flow picture added. 2497 o Section on privacy considerations expanded. 2499 14.7. Changes between -00 and -01 2501 o Add the SCT feedback mechanism: Clients send SCTs to originating 2502 web server which shares them with auditors. 2504 o Stop assuming that clients see STHs. 2506 o Don't use HTTP headers but instead .well-known URL's - avoid that 2507 battle. 2509 o Stop referring to trans-gossip and trans-gossip-transport-https - 2510 too complicated. 2512 o Remove all protocols but HTTPS in order to simplify - let's come 2513 back and add more later. 2515 o Add more reasoning about privacy. 2517 o Do specify data formats. 2519 15. References 2521 15.1. Normative References 2523 [RFC-6962-BIS-27] 2524 Laurie, B., Langley, A., Kasper, E., Messeri, E., and R. 2525 Stradling, "Certificate Transparency", October 2017, 2526 . 2529 [RFC6962] Laurie, B., Langley, A., and E. Kasper, "Certificate 2530 Transparency", RFC 6962, DOI 10.17487/RFC6962, June 2013, 2531 . 2533 [RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 2534 Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2535 2014, . 2537 [RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext 2538 Transfer Protocol Version 2 (HTTP/2)", RFC 7540, 2539 DOI 10.17487/RFC7540, May 2015, . 2542 15.2. Informative References 2544 [double-keying] 2545 Perry, M., Clark, E., and S. Murdoch, "Cross-Origin 2546 Identifier Unlinkability", May 2015, 2547 . 2550 [draft-ct-over-dns] 2551 Laurie, B., Phaneuf, P., and A. Eijdenberg, "Certificate 2552 Transparency over DNS", February 2016, 2553 . 2556 [draft-ietf-trans-threat-analysis-12] 2557 Kent, S., "Attack and Threat Model for Certificate 2558 Transparency", October 2017, 2559 . 2562 [dual-ca-compromise-attack] 2563 Gillmor, D., "can CT defend against dual CA compromise?", 2564 n.d., . 2567 [gossip-mixing] 2568 Ritter, T., "A Bit on Certificate Transparency Gossip", 2569 June 2016, . 2572 [trickle] Serjantov, A., Dingledine, R., and . Paul Syverson, "From 2573 a Trickle to a Flood: Active Attacks on Several Mix 2574 Types", October 2002, 2575 . 2577 Authors' Addresses 2579 Linus Nordberg 2580 NORDUnet 2582 Email: linus@nordu.net 2584 Daniel Kahn Gillmor 2585 ACLU 2587 Email: dkg@fifthhorseman.net 2589 Tom Ritter 2591 Email: tom@ritter.vg