idnits 2.17.00 (12 Aug 2021) /tmp/idnits16666/draft-ietf-dnsop-serve-stale-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with non-RFC2606-compliant FQDNs in the document. -- The draft header indicates that this document updates RFC1034, but the abstract doesn't seem to directly say this. It does mention RFC1034 though, so this could be OK. -- The draft header indicates that this document updates RFC1035, but the abstract doesn't seem to directly say this. It does mention RFC1035 though, so this could be OK. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year (Using the creation date from RFC2181, updated by this document, for RFC5378 checks: 1996-02-20) -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (August 30, 2019) is 994 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DNSOP Working Group D. Lawrence 3 Internet-Draft Oracle 4 Updates: 1034, 1035, 2181 (if approved) W. Kumari 5 Intended status: Standards Track P. Sood 6 Expires: March 2, 2020 Google 7 August 30, 2019 9 Serving Stale Data to Improve DNS Resiliency 10 draft-ietf-dnsop-serve-stale-07 12 Abstract 14 This draft defines a method (serve-stale) for recursive resolvers to 15 use stale DNS data to avoid outages when authoritative nameservers 16 cannot be reached to refresh expired data. One of the motivations 17 for serve-stale is to make the DNS more resilient to DoS attacks, and 18 thereby make them less attractive as an attack vector. This document 19 updates the definitions of TTL from RFC 1034 and RFC 1035 so that 20 data can be kept in the cache beyond the TTL expiry, and also updates 21 RFC 2181 by interpreting values with the high order bit set as being 22 positive, rather than 0, and also suggests a cap of 7 days. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on March 2, 2020. 41 Copyright Notice 43 Copyright (c) 2019 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 4. Standards Action . . . . . . . . . . . . . . . . . . . . . . 4 62 5. Example Method . . . . . . . . . . . . . . . . . . . . . . . 4 63 6. Implementation Considerations . . . . . . . . . . . . . . . . 6 64 7. Implementation Caveats . . . . . . . . . . . . . . . . . . . 8 65 8. Implementation Status . . . . . . . . . . . . . . . . . . . . 9 66 9. EDNS Option . . . . . . . . . . . . . . . . . . . . . . . . . 9 67 10. Security Considerations . . . . . . . . . . . . . . . . . . . 10 68 11. Privacy Considerations . . . . . . . . . . . . . . . . . . . 10 69 12. NAT Considerations . . . . . . . . . . . . . . . . . . . . . 10 70 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 71 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 72 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 73 15.1. Normative References . . . . . . . . . . . . . . . . . . 11 74 15.2. Informative References . . . . . . . . . . . . . . . . . 11 75 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 77 1. Introduction 79 Traditionally the Time To Live (TTL) of a DNS resource record has 80 been understood to represent the maximum number of seconds that a 81 record can be used before it must be discarded, based on its 82 description and usage in [RFC1035] and clarifications in [RFC2181]. 84 This document proposes that the definition of the TTL be explicitly 85 expanded to allow for expired data to be used in the exceptional 86 circumstance that a recursive resolver is unable to refresh the 87 information. It is predicated on the observation that authoritative 88 answer unavailability can cause outages even when the underlying data 89 those servers would return is typically unchanged. 91 We describe a method below for this use of stale data, balancing the 92 competing needs of resiliency and freshness. 94 This document updates the definitions of TTL from [RFC1034] and 95 [RFC1035] so that data can be kept in the cache beyond the TTL 96 expiry, and also updates [RFC2181] by interpreting values with the 97 high order bit set as being positive, rather than 0, and also 98 suggests a cap of 7 days. 100 2. Terminology 102 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 103 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 104 "OPTIONAL" in this document are to be interpreted as described in BCP 105 14 [RFC2119] [RFC8174] when, and only when, they appear in all 106 capitals, as shown here. 108 For a comprehensive treatment of DNS terms, please see [RFC8499]. 110 3. Background 112 There are a number of reasons why an authoritative server may become 113 unreachable, including Denial of Service (DoS) attacks, network 114 issues, and so on. If a recursive server is unable to contact the 115 authoritative servers for a query but still has relevant data that 116 has aged past its TTL, that information can still be useful for 117 generating an answer under the metaphorical assumption that "stale 118 bread is better than no bread." 120 [RFC1035] Section 3.2.1 says that the TTL "specifies the time 121 interval that the resource record may be cached before the source of 122 the information should again be consulted", and Section 4.1.3 further 123 says the TTL, "specifies the time interval (in seconds) that the 124 resource record may be cached before it should be discarded." 126 A natural English interpretation of these remarks would seem to be 127 clear enough that records past their TTL expiration must not be used. 128 However, [RFC1035] predates the more rigorous terminology of 129 [RFC2119] which softened the interpretation of "may" and "should". 131 [RFC2181] aimed to provide "the precise definition of the Time to 132 Live", but in Section 8 was mostly concerned with the numeric range 133 of values and the possibility that very large values should be 134 capped. (It also has the curious suggestion that a value in the 135 range 2147483648 to 4294967295 should be treated as zero.) It closes 136 that section by noting, "The TTL specifies a maximum time to live, 137 not a mandatory time to live." This is again not [RFC2119]-normative 138 language, but does convey the natural language connotation that data 139 becomes unusable past TTL expiry. 141 Several recursive resolver operators currently use stale data for 142 answers in some way, including Akamai. A number of recursive 143 resolver packages (including BIND, Know, OpenDNS, Unbound) provide 144 options to use stale data. Apple MacOS can also use stale data as 145 part of the Happy Eyeballs algorithms in mDNSResponder. The 146 collective operational experience is that it provides significant 147 benefit with minimal downside. 149 4. Standards Action 151 The definition of TTL in [RFC1035] Sections 3.2.1 and 4.1.3 is 152 amended to read: 154 TTL a 32-bit unsigned integer number of seconds that specifies the 155 duration that the resource record MAY be cached before the source 156 of the information MUST again be consulted. Zero values are 157 interpreted to mean that the RR can only be used for the 158 transaction in progress, and should not be cached. Values SHOULD 159 be capped on the orders of days to weeks, with a recommended cap 160 of 604,800 seconds (seven days). If the data is unable to be 161 authoritatively refreshed when the TTL expires, the record MAY be 162 used as though it is unexpired. 164 Interpreting values which have the high order bit set as being 165 positive, rather than 0, is a change from [RFC2181]. Suggesting a 166 cap of seven days, rather than the 68 years allowed by [RFC2181], 167 reflects the current practice of major modern DNS resolvers. 169 When returning a response containing stale records, a recursive 170 resolver MUST set the TTL of each expired record in the message to a 171 value greater than 0, with 30 seconds RECOMMENDED. 173 Answers from authoritative servers that have a DNS Response Code of 174 either 0 (NoError) or 3 (NXDomain) and the Authoritative Answers (AA) 175 bit set MUST be considered to have refreshed the data at the 176 resolver. Answers from authoritative servers that have any other 177 response code SHOULD be considered a failure to refresh the data and 178 therefor leave any previous state intact. 180 5. Example Method 182 There is more than one way a recursive resolver could responsibly 183 implement this resiliency feature while still respecting the intent 184 of the TTL as a signal for when data is to be refreshed. 186 In this example method four notable timers drive considerations for 187 the use of stale data: 189 o A client response timer, which is the maximum amount of time a 190 recursive resolver should allow between the receipt of a 191 resolution request and sending its response. 193 o A query resolution timer, which caps the total amount of time a 194 recursive resolver spends processing the query. 196 o A failure recheck timer, which limits the frequency at which a 197 failed lookup will be attempted again. 199 o A maximum stale timer, which caps the amount of time that records 200 will be kept past their expiration. 202 Most recursive resolvers already have the query resolution timer, and 203 effectively some kind of failure recheck timer. The client response 204 timer and maximum stale timer are new concepts for this mechanism. 206 When a request is received by a recursive resolver, it should start 207 the client response timer. This timer is used to avoid client 208 timeouts. It should be configurable, with a recommended value of 1.8 209 seconds as being just under a common timeout value of 2 seconds while 210 still giving the resolver a fair shot at resolving the name. 212 The resolver then checks its cache for any unexpired records that 213 satisfy the request and returns them if available. If it finds no 214 relevant unexpired data and the Recursion Desired flag is not set in 215 the request, it should immediately return the response without 216 consulting the cache for expired records. Typically this response 217 would be a referral to authoritative nameservers covering the zone, 218 but the specifics are implementation dependent. 220 If iterative lookups will be done, then the failure recheck timer is 221 consulted. Attempts to refresh from non-responsive or otherwise 222 failing authoritative nameservers are recommended to be done no more 223 frequently than every 30 seconds. If this request was received 224 within this period, the cache may be immediately consulted for stale 225 data to satisfy the request. 227 Outside the period of the failure recheck timer, the resolver should 228 start the query resolution timer and begin the iterative resolution 229 process. This timer bounds the work done by the resolver when 230 contacting external authorities, and is commonly around 10 to 30 231 seconds. If this timer expires on an attempted lookup that is still 232 being processed, the resolution effort is abandoned. 234 If the answer has not been completely determined by the time the 235 client response timer has elapsed, the resolver should then check its 236 cache to see whether there is expired data that would satisfy the 237 request. If so, it adds that data to the response message with a TTL 238 greater than 0 (as specified in Section 4). The response is then 239 sent to the client while the resolver continues its attempt to 240 refresh the data. 242 When no authorities are able to be reached during a resolution 243 attempt, the resolver should attempt to refresh the delegation and 244 restart the iterative lookup process with the remaining time on the 245 query resolution timer. This resumption should be done only once 246 during one resolution effort. 248 Outside the resolution process, the maximum stale timer is used for 249 cache management and is independent of the query resolution process. 250 This timer is conceptually different from the maximum cache TTL that 251 exists in many resolvers, the latter being a clamp on the value of 252 TTLs as received from authoritative servers and recommended to be 253 seven days in the TTL definition in Section 4. The maximum stale 254 timer should be configurable, and defines the length of time after a 255 record expires that it should be retained in the cache. The 256 suggested value is between 1 and 3 days. 258 6. Implementation Considerations 260 This document mainly describes the issues behind serving stale data 261 and intentionally does not provide a formal algorithm. The concept 262 is not overly complex, and the details are best left to resolver 263 authors to implement in their codebases. The processing of serve- 264 stale is a local operation, and consistent variables between 265 deployments are not needed for interoperability. However, we would 266 like to highlight the impact of various implementation choices, 267 starting with the timers involved. 269 The most obvious of these is the maximum stale timer. If this 270 variable is too large it could cause excessive cache memory usage, 271 but if it is too small, the serve-stale technique becomes less 272 effective, as the record may not be in the cache to be used if 273 needed. Shorter values, even less than a day, can effectively handle 274 the vast majority of outages. Longer values, as much as a week, give 275 time for monitoring systems to notice a resolution problem and for 276 human intervention to fix it; operational experience has been that 277 sometimes the right people can be hard to track down and 278 unfortunately slow to remedy the situation. 280 Increased memory consumption could be mitigated by prioritizing 281 removal of stale records over non-expired records during cache 282 exhaustion. Implementations may also wish to consider whether to 283 track the names in requests for their last time of use or their 284 popularity, using that as an additional factor when considering cache 285 eviction. A feature to manually flush only stale records could also 286 be useful. 288 The client response timer is another variable which deserves 289 consideration. If this value is too short, there exists the risk 290 that stale answers may be used even when the authoritative server is 291 actually reachable but slow; this may result in sub-optimal answers 292 being returned. Conversely, waiting too long will negatively impact 293 user experience. 295 The balance for the failure recheck timer is responsiveness in 296 detecting the renewed availability of authorities versus the extra 297 resource use for resolution. If this variable is set too large, 298 stale answers may continue to be returned even after the 299 authoritative server is reachable; per [RFC2308], Section 7, this 300 should be no more than five minutes. If this variable is too small, 301 authoritative servers may be rapidly hit with a significant amount of 302 traffic when they become reachable again. 304 Regarding the TTL to set on stale records in the response, 305 historically TTLs of zero seconds have been problematic for some 306 implementations, and negative values can't effectively be 307 communicated to existing software. Other very short TTLs could lead 308 to congestive collapse as TTL-respecting clients rapidly try to 309 refresh. The recommended value of 30 seconds not only sidesteps 310 those potential problems with no practical negative consequences, it 311 also rate limits further queries from any client that honors the TTL, 312 such as a forwarding resolver. 314 Another implementation consideration is the use of stale nameserver 315 addresses for lookups. This is mentioned explicitly because, in some 316 resolvers, getting the addresses for nameservers is a separate path 317 from a normal cache lookup. If authoritative server addresses are 318 not able to be refreshed, resolution can possibly still be successful 319 if the authoritative servers themselves are up. For instance, 320 consider an attack on a top-level domain that takes its nameservers 321 offline; serve-stale resolvers that had expired glue addresses for 322 subdomains within that TLD would still be able to resolve names 323 within those subdomains, even those it had not previously looked up. 325 The directive in Section 4 that only NoError and NXDomain responses 326 should invalidate any previously associated answer stems from the 327 fact that no other RCODEs which a resolver normally encounters makes 328 any assertions regarding the name in the question or any data 329 associated with it. This comports with existing resolver behavior 330 where a failed lookup (say, during pre-fetching) doesn't impact the 331 existing cache state. Some authoritative servers operators have said 332 that they would prefer stale answers to be used in the event that 333 their servers are responding with errors like ServFail instead of 334 giving true authoritative answers. Implementers MAY decide to return 335 stale answers in this situation. 337 Since the goal of serve-stale is to provide resiliency for all 338 obvious errors to refresh data, these other RCODEs are treated as 339 though they are equivalent to not getting an authoritative response. 340 Although NXDomain for a previously existing name might well be an 341 error, it is not handled that way because there is no effective way 342 to distinguish operator intent for legitimate cases versus error 343 cases. 345 During discussion in the IETF, it was suggested that, if all 346 authorities return responses with RCODE of Refused, it may be an 347 explicit signal to take down the zone from servers that still have 348 the zone's delegation pointed to them. Refused, however, is also 349 overloaded to mean multiple possible failures which could represent 350 transient configuration failures. Operational experience has shown 351 that purposely returning Refused is a poor way to achieve an explicit 352 takedown of a zone compared to either updating the delegation or 353 returning NXDomain with a suitable SOA for extended negative caching. 354 Implementers MAY nonetheless consider whether to treat all 355 authorities returning Refused as preempting the use of stale data. 357 7. Implementation Caveats 359 Stale data is used only when refreshing has failed in order to adhere 360 to the original intent of the design of the DNS and the behaviour 361 expected by operators. If stale data were to always be used 362 immediately and then a cache refresh attempted after the client 363 response has been sent, the resolver would frequently be sending data 364 that it would have had no trouble refreshing. Because modern 365 resolvers use techniques like pre-fetching and request coalescing for 366 efficiency, it is not necessary that every client request needs to 367 trigger a new lookup flow in the presence of stale data, but rather 368 that a good-faith effort has been recently made to refresh the stale 369 data before it is delivered to any client. 371 It is important to continue the resolution attempt after the stale 372 response has been sent, until the query resolution timeout, because 373 some pathological resolutions can take many seconds to succeed as 374 they cope with unavailable servers, bad networks, and other problems. 375 Stopping the resolution attempt when the response with expired data 376 has been sent would mean that answers in these pathological cases 377 would never be refreshed. 379 The continuing prohibition against using data with a 0 second TTL 380 beyond the current transaction explicitly extends to it being 381 unusable even for stale fallback, as it is not to be cached at all. 383 Be aware that Canonical Name (CNAME) and DNAME [RFC6672] records 384 mingled in the expired cache with other records at the same owner 385 name can cause surprising results. This was observed with an initial 386 implementation in BIND when a hostname changed from having an IPv4 387 Address (A) record to a CNAME. The version of BIND being used did 388 not evict other types in the cache when a CNAME was received, which 389 in normal operations is not a significant issue. However, after both 390 records expired and the authorities became unavailable, the fallback 391 to stale answers returned the older A instead of the newer CNAME. 393 8. Implementation Status 395 [RFC Editor: per RFC 6982 this section should be removed prior to 396 publication.] 398 The algorithm described in Section 5 was originally implemented as a 399 patch to BIND 9.7.0. It has been in production on Akamai's 400 production network since 2011, and effectively smoothed over 401 transient failures and longer outages that would have resulted in 402 major incidents. The patch was contributed to Internet Systems 403 Consortium and the functionality is now available in BIND 9.12 via 404 the options stale-answer-enable, stale-answer-ttl, and max-stale-ttl. 406 Unbound has a similar feature for serving stale answers, but will 407 respond with stale data immediately if it has recently tried and 408 failed to refresh the answer by pre-fetching. 410 Knot Resolver has a demo module here: https://knot- 411 resolver.readthedocs.io/en/stable/modules.html#serve-stale 413 Details of Apple's implementation are not currently known. 415 In the research paper "When the Dike Breaks: Dissecting DNS Defenses 416 During DDoS" [DikeBreaks], the authors detected some use of stale 417 answers by resolvers when authorities came under attack. Their 418 research results suggest that more widespread adoption of the 419 technique would significantly improve resiliency for the large number 420 of requests that fail or experience abnormally long resolution times 421 during an attack. 423 9. EDNS Option 425 During the discussion of serve-stale in the IETF, it was suggested 426 that an EDNS option should be available to either explicitly opt-in 427 to getting data that is possibly stale, or at least as a debugging 428 tool to indicate when stale data has been used for a response. 430 The opt-in use case was rejected as the technique was meant to be 431 immediately useful in improving DNS resiliency for all clients. 433 The reporting case was ultimately also rejected because even the 434 simpler version of a proposed option was still too much bother to 435 implement for too little perceived value. 437 10. Security Considerations 439 The most obvious security issue is the increased likelihood of DNSSEC 440 validation failures when using stale data because signatures could be 441 returned outside their validity period. This would only be an issue 442 if the authoritative servers are unreachable, the only time the 443 techniques in this document are used, and thus does not introduce a 444 new failure in place of what would have otherwise been success. 446 Additionally, bad actors have been known to use DNS caches to keep 447 records alive even after their authorities have gone away. This 448 potentially makes that easier, although without introducing a new 449 risk. 451 In [CloudStrife], it was demonstrated how stale DNS data, namely 452 hostnames pointing to addresses that are no longer in use by the 453 owner of the name, can be used to co-opt security such as to get 454 domain-validated certificates fraudulently issued to an attacker. 455 While this document does not create a new vulnerability in this area, 456 it does potentially enlarge the window in which such an attack could 457 be made. A proposed mitigation is that certificate authorities 458 should fully look up each name starting at the DNS root for every 459 name lookup. Alternatively, CAs should use a resolver that is not 460 serving stale data. 462 11. Privacy Considerations 464 This document does not add any practical new privacy issues. 466 12. NAT Considerations 468 The method described here is not affected by the use of NAT devices. 470 13. IANA Considerations 472 There are no IANA considerations. 474 14. Acknowledgements 476 The authors wish to thank Robert Edmonds, Tony Finch, Bob Harold, 477 Tatuya Jinmei, Matti Klock, Jason Moreau, Giovane Moura, Jean Roy, 478 Mukund Sivaraman, Davey Song, Paul Vixie, Ralf Weber and Paul Wouters 479 for their review and feedback. 481 Paul Hoffman deserves special thanks for submitting a number of Pull 482 Requests. 484 15. References 486 15.1. Normative References 488 [RFC1034] Mockapetris, P., "Domain names - concepts and facilities", 489 STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987, 490 . 492 [RFC1035] Mockapetris, P., "Domain names - implementation and 493 specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, 494 November 1987, . 496 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 497 Requirement Levels", BCP 14, RFC 2119, 498 DOI 10.17487/RFC2119, March 1997, 499 . 501 [RFC2181] Elz, R. and R. Bush, "Clarifications to the DNS 502 Specification", RFC 2181, DOI 10.17487/RFC2181, July 1997, 503 . 505 [RFC2308] Andrews, M., "Negative Caching of DNS Queries (DNS 506 NCACHE)", RFC 2308, DOI 10.17487/RFC2308, March 1998, 507 . 509 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 510 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 511 May 2017, . 513 15.2. Informative References 515 [CloudStrife] 516 Borgolte, K., Fiebig, T., Hao, S., Kruegel, C., and G. 517 Vigna, "Cloud Strife: Mitigating the Security Risks of 518 Domain-Validated Certificates", ACM 2018 Applied 519 Networking Research Workshop, DOI 10.1145/3232755.3232859, 520 July 2018, . 524 [DikeBreaks] 525 Moura, G., Heidemann, J., Mueller, M., Schmidt, R., and M. 526 Davids, "When the Dike Breaks: Dissecting DNS Defenses 527 During DDos", ACM 2018 Internet Measurement Conference, 528 DOI 10.1145/3278532.3278534, October 2018, 529 . 531 [RFC6672] Rose, S. and W. Wijngaards, "DNAME Redirection in the 532 DNS", RFC 6672, DOI 10.17487/RFC6672, June 2012, 533 . 535 [RFC8499] Hoffman, P., Sullivan, A., and K. Fujiwara, "DNS 536 Terminology", BCP 219, RFC 8499, DOI 10.17487/RFC8499, 537 January 2019, . 539 Authors' Addresses 541 David C Lawrence 542 Oracle 544 Email: tale@dd.org 546 Warren "Ace" Kumari 547 Google 548 1600 Amphitheatre Parkway 549 Mountain View CA 94043 550 USA 552 Email: warren@kumari.net 554 Puneet Sood 555 Google 557 Email: puneets@google.com