idnits 2.17.00 (12 Aug 2021) /tmp/idnits56162/draft-ietf-mediaman-haptics-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (19 November 2021) is 176 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MEDIAMAN Y. K. Muthusamy 3 Internet-Draft C. Ullrich 4 Intended status: Standards Track Immersion Corporation 5 Expires: 23 May 2022 19 November 2021 7 The 'haptics' Top-level Media Type 8 draft-ietf-mediaman-haptics-00 10 Abstract 12 This memo serves to register and document the 'haptics' top-level 13 media type, under which subtypes for representation formats for 14 haptics may be registered. This document also serves as a 15 registration application for a set of intended subtypes, which are 16 representative of some existing subtypes already in use. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on 23 May 2022. 35 Copyright Notice 37 Copyright (c) 2021 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 42 license-info) in effect on the date of publication of this document. 43 Please review these documents carefully, as they describe your rights 44 and restrictions with respect to this document. Code Components 45 extracted from this document must include Revised BSD License text as 46 described in Section 4.e of the Trust Legal Provisions and are 47 provided without warranty as described in the Revised BSD License. 49 RFC XXXX The 'haptics' Top-level Media Type November 2021 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Background and Justification . . . . . . . . . . . . . . . . 3 56 2.1. MPEG ISOBMFF . . . . . . . . . . . . . . . . . . . . . . 3 57 2.2. Haptic Sub-modalities . . . . . . . . . . . . . . . . . . 4 58 2.3. Another Human Sense . . . . . . . . . . . . . . . . . . . 4 59 2.4. Commercial Uptake . . . . . . . . . . . . . . . . . . . . 4 60 2.5. Haptic Data Formats in Use . . . . . . . . . . . . . . . 5 61 2.6. Haptic Subtypes (envisioned standards) . . . . . . . . . 6 62 2.7. 'application' top-level type not suitable . . . . . . . . 6 63 3. Security Considerations . . . . . . . . . . . . . . . . . . . 7 64 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 65 4.1. Definition and Encoding . . . . . . . . . . . . . . . . . 8 66 4.2. Registration Procedure . . . . . . . . . . . . . . . . . 8 67 4.3. Subtype Registrations . . . . . . . . . . . . . . . . . . 8 68 4.3.1. IVS Haptics Type . . . . . . . . . . . . . . . . . . 8 69 4.3.2. HAPT Haptics Type . . . . . . . . . . . . . . . . . . 9 70 5. Normative References . . . . . . . . . . . . . . . . . . . . 10 71 6. Informative References . . . . . . . . . . . . . . . . . . . 10 72 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 74 1. Introduction 76 The term 'haptics' refers to the generation of touch-related 77 sensations in a device or interface. Haptics is widely used in 78 consumer devices in order to provide touch-based feedback to users. 79 The most common use of haptics is in mobile devices, where it is used 80 to provide feedback to users interacting with the touchscreen, e.g., 81 typing on a virtual keyboard. Haptic technologies are unlike audio 82 and visual enabling technologies in the sense that they require some 83 form of actuation in order to create a tactile sensation. For mobile 84 phones and game controllers, these actuators are typically small 85 vibrating motors. For large touchscreens in vehicles, these 86 actuators can be specialized piezoelectric materials. Haptic 87 capabilities are found in nearly every modern smartphone and game and 88 virtual reality controller, making these devices an ideal target for 89 enhanced media experiences. 91 Internet Media Types [RFC6838] are used to label content carried over 92 Internet protocols. This document defines a new top-level type 93 'haptics' according to Section 4.2.7 of [RFC6838]. This top-level 94 type indicates that the content specifies haptic data. Under this 95 top-level type, different representation formats of haptics may be 96 registered. 98 RFC XXXX The 'haptics' Top-level Media Type November 2021 100 1.1. Terminology 102 The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, 103 SHOULD,SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear 104 in this document, are to be interpreted as described in [RFC2119]. 106 2. Background and Justification 108 Haptic signals provide an additional layer of entertainment and 109 sensory immersion for the user. Haptic tracks, in separate files, 110 can be combined with audio/video files and played back in sync to 111 provide an overall immersive media experience (audio, visual, 112 tactile) for the user. More recently, haptic tracks embedded in 113 standard file formats such as ISOBMFF (ISO Base Media File Format), 114 enable playback of the haptic signals over one or more actuators, 115 simultaneously with audio and video playback. Haptic signals are 116 also part of media streams that use RTP, such as those for streaming 117 games, XR, and wearables. 119 2.1. MPEG ISOBMFF 121 Historically, there has not been a registration of formats for 122 haptics. However, haptics has been proposed as a first-order media 123 type (at the same level as audio and video) in ISOBMFF. This 124 proposal was made to the MPEG Systems File Format sub-group in April 125 2020. The proposal was accepted and has since progressed to 126 International Standard, to be published in December 2021 127 [ISOBMFF-IS]. Haptics is officially part of the ISO/IEC 14496-12 128 (ISOBMFF) standard, 7th Edition. Given this development, a strong 129 case can be made for haptics to be added to the list of top-level 130 media types recognized by the IETF. 132 We envision the following designations for haptics in mp4 files, once 133 the top-level type 'haptics' is registered: 135 1. 'haptics/mp4' - mp4 files with just haptic tracks and no audio or 136 video in them (e.g., streaming games, haptics files for haptic 137 vests, belts, gloves, etc.) 139 2. 'video/mp4' - mp4 files with video, audio, and haptics (to ensure 140 consistency with existing mp4 files with video content) 142 3. 'audio/mp4' - mp4 files with audio and haptics (to ensure 143 consistency with existing mp4 files with audio content without 144 any video) 146 RFC XXXX The 'haptics' Top-level Media Type November 2021 148 2.2. Haptic Sub-modalities 150 There are multiple sub-modalities of haptics: 152 * Vibrotactile (touch, vibration) 154 * Kinesthetic (force feedback) 156 * Surface (surface friction) 158 * Spatial, non-contact (ultrasound) 160 * Thermal (temperature) 162 Therefore, designating 'haptics' as a top-level media type would 163 enable the definition of data formats pertaining to these sub- 164 modalities in a more streamlined manner. This would not be possible 165 if 'haptics' were to be placed under other top-level types like 166 'audio', 'video', or 'application'. 168 2.3. Another Human Sense 170 The top-level media type 'audio' pertains to the human sense of 171 hearing, the top-level media type 'video' pertains to the human sense 172 of seeing, so it only makes sense for the (equally important) human 173 sense of touch to be represented by another top-level media type 174 'haptics'. Placing 'haptics' under 'audio' or 'video' is not 175 reflective of the kinds of files or use cases that would need haptics 176 but have nothing whatsoever to do with audio or video. 178 2.4. Commercial Uptake 180 Haptics is rapidly becoming a standard feature of consumer electronic 181 devices. For example: 183 * iPhone (206+ million units sold in 2020): native support for 184 haptic encoded data 186 * Android (1.38+ billion units sold in 2020): API support of haptic 187 buffers 189 * W3C (HTML vibration API [W3C-Vibration]): Optionally supported in 190 mobile web browsers. W3C has also defined vibration extensions 191 for gamepads [W3C-Gamepad] 193 * Game consoles (39+ million units sold in 2019): MS Xbox, Sony 194 PlayStation, Nintendo Switch, etc. 196 RFC XXXX The 'haptics' Top-level Media Type November 2021 198 * XR devices (9+ million units sold in 2019): OpenXR haptic API 200 Haptic media is expected to be commonly exchanged between these 201 devices. Since they represent the majority of CE devices, a strong 202 case can be made for 'haptics' as a top-level media type. 204 2.5. Haptic Data Formats in Use 206 There are multiple instances of existing haptic data formats that 207 would live as sub-types under the proposed 'haptics' top-level media 208 type. While these subtypes have *not* been registered with IANA or 209 standardized (yet), the prevalence of these haptic data formats in a 210 large number of devices around the world, pre-dating the 211 standardization of haptic tracks in ISOBMFF, provides a compelling 212 argument for 'haptics' to be designated as a top-level media type: 214 * 'ahap': The AHAP haptic data format [AHAP] is currently the 215 standard encoding on all iOS devices + iOS connected game 216 peripherals. The format has seen usage and adoption beyond Apple 217 devices as well, with decoders available for Android and other XR 218 systems. 220 * 'ogg': Google has introduced a proprietary extension to the OGG 221 format in the latest version of Android 11. This encoding enables 222 haptic media to be stored in OGG files. 224 * 'ivs': The IVS haptic data format is currently a vendor-specific 225 format that is in use: 227 - In mobile phones from LG Electronics (specifically, the models 228 V30, V40, and the newest V50) that are sold worldwide 230 - In gaming phones from ASUS (specifically, models ROG, ROG Phone 231 II, ROG Phone 3) that are sold worldwide 233 * 'hapt': The HAPT haptic data format is currently a vendor-specific 234 format that is in use: 236 - In mobile haptic advertising (for W3C devices) 238 - The following Japanese game developers use the HAPT format as 239 part of Immersion's TouchSense SDK: 241 o KLAB: https://www.klab.com/en/ (https://www.klab.com/en/) 243 o Craft&Meister: http://www.crafts-meister.co.jp/pc/ 244 company_en.html (http://www.crafts-meister.co.jp/pc/ 245 company_en.html) 247 RFC XXXX The 'haptics' Top-level Media Type November 2021 249 - Tencent is using the TouchSense SDK for their popular social 250 media application QQ and live streaming application NOW: 251 Immersion-Announces-Tencent-Licenses-TouchSense-Technology- 252 Deliver 253 (https://www.businesswire.com/news/home/20171026006443/en/ 254 Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE- 255 Technology-Deliver) 257 Given the widespread use of these subtypes, it makes sense for 258 'haptics' to be a top-level media type. 260 2.6. Haptic Subtypes (envisioned standards) 262 The MPEG ISOBMFF proposal included an informative annex of known 263 haptic coding formats with proposed FourCC codes for them. These 264 codes are not registered yet, but the plan is indeed to standardize 265 these haptic coding formats in the near future. Once standardized, 266 they will also live as subtypes under the proposed 'haptics' top- 267 level media type: 269 * 'hmpg': the selected coding format from the MPEG Call for 270 Proposals on the Coded Representation of Haptics 271 [MPEG-Haptics-CfP] 273 * 'hiee': IEEE P1918.1.1 vibrotactile coding standard [IEEE-P191811] 274 being developed under the IEEE Tactile Internet initiative as part 275 of the 5G URLL profile 277 * 'henm': enumerated effects haptic coding format (based on MIDI) 279 * 'havc': audio-to-vibe haptic coding format (automatic audio to 280 vibration conversion algorithms) 282 2.7. 'application' top-level type not suitable 284 From the above arguments, it is clear that haptics does not really 285 belong under any other media type. To reiterate, there are three 286 main reasons why the 'haptics' media type does not fit under the 287 'application' top-level type: 289 * haptics connects to a sensory system, touch/motion, directly, and 290 is more specific than the abstract 'application' type, and 292 * 'application' has historically been used for applications, i.e., 293 code, which means it is viewed and treated with great care for 294 security. 'haptics' is not code, just as 'audio' and 'video' are 295 not code either. 297 RFC XXXX The 'haptics' Top-level Media Type November 2021 299 * haptics is a property of a media stream, it is not an application 300 under any normal definition. As such, it should be its own type. 302 3. Security Considerations 304 Haptics are interpreted data structures that represent collections of 305 different media rendering instructions intended to be decoded and 306 rendered on target device hardware. Haptic data can be represented 307 as collections of signal data and/or descriptive text in XML/JSON or 308 similar format. Signal data is typically not executed by endpoint 309 processors and represents minimal security risk. Descriptive text is 310 typically parsed and represented in memory using standard XML data 311 structures. This data is utilized to construct one or more signals 312 that are sent to the endpoint device hardware. 314 Because of the media/rendering nature of the data path for haptic 315 coded data the security profile of haptic data is expected to be 316 largely consistent with the security profile of visual and audio 317 media data. 319 As with any synthesized media data (audio, video, and haptics), there 320 is a security risk associated with execution of commands based on the 321 descriptive encoding either through its inherent extensibility or 322 through the insertion of arbitrary executable data in the descriptive 323 format itself. Indeed, media rendering systems are normally 324 implemented with a mix of user and kernel space execution since these 325 media must ultimately make their way to a hardware system. In 326 theory, malicious instructions present in descriptive haptic media 327 have the potential to execute arbitrary code in kernel space, 328 effectively bypassing system permissions structures and/or execution 329 sandboxes. 331 Haptics, audio, and video media have widespread use and careful 332 attention should be paid by operating system and device driver 333 implementors to ensure that synthesis and rendering signal paths do 334 not provide attack surfaces for malicious payloads. Ultimately, any 335 coded representation of haptic media is insufficient to implicitly 336 provide sufficient security and this protection should be enforced by 337 the operating system implementor. 339 4. IANA Considerations 341 This specification registers a new top-level type, 'haptics', in the 342 standards tree, adds it as an alternative value of "Type Name" in the 343 media types registration form [Media-Type-Registration], and 344 registers several subtypes for it. 346 RFC XXXX The 'haptics' Top-level Media Type November 2021 348 4.1. Definition and Encoding 350 'haptics' as the primary media content type indicates that the 351 content identified by it requires a certain haptics subsystem such as 352 low-level haptics APIs, which in turn will require hardware 353 capabilities such as one or more actuators to render the haptics 354 media. The 'haptics' media type does not provide any specific 355 information about the underlying data format and how the haptics 356 information should be interpreted -- the subtypes defined within a 357 'haptics' tree name the specific haptic formats. Unrecognized 358 subtypes of 'haptics' should be treated as 'application/octet- 359 stream'. Implementations may still pass unrecognized subtypes to the 360 haptics subsystem and associated rendering hardware. 362 4.2. Registration Procedure 364 New haptics formats should be registered using the online form 365 [Media-Type-Registration]. [RFC6838] should be consulted on 366 registration procedures. In particular, the haptics specification 367 should preferably be freely available. 369 Note that new parameter sub-values may be defined in the future. If 370 an implementation does not recognize a sub-value in the comma- 371 separated list, it should ignore the sub-value and continue 372 processing the other sub-values in the list. 374 4.3. Subtype Registrations 376 In this section, the initial entries under the top-level 'haptics' 377 media type are specified. They also serve as examples for future 378 registrations. 380 4.3.1. IVS Haptics Type 382 Type name: haptics 384 Subtype name: ivs 386 Required parameters: None 388 Optional parameters: None 390 Encoding considerations: Text/binary 392 Interoperability considerations: The IVS format is a device- 393 independent haptic effect coding. It is designed to enable 394 interoperability between distinct physical endpoints. Not all 395 devices may be able to render all effects present in an IVS file. 397 RFC XXXX The 'haptics' Top-level Media Type November 2021 399 Published specification: ISO/IEC JTC 1/SC 29/WG 2 N 72 "Encoder Input 400 Format for MPEG Haptics" being developed by ISO/IEC JTC1/SC29 WG 2 401 [MPEG-Haptics-Encoder]. 403 Applications that use this media type: All applications that are able 404 to create, edit, or display haptic media content. 406 Additional information: 408 * File extension(s): Haptic file extensions used for IVS files: .ivs 409 (xml) and .ivt (binary) 411 * Macintosh file type code(s): (no code specified) 413 * Macintosh Universal Type Identifier code: None 415 * Fragment Identifier: None 417 * Deprecated Alias: None 419 Person & email address to contact for further information: Yeshwant 420 Muthusamy(ymuthusamy@immersion.com) 422 Change controller: Immersion Corporation 424 4.3.2. HAPT Haptics Type 426 Type name: haptics 428 Subtype name: hapt 430 Required parameters: None 432 Optional parameters: None 434 Encoding considerations: Text/binary 436 Interoperability considerations: The HAPT format is a device- 437 dependent haptic effect coding based on the RIFF coding standard. It 438 is designed to enable efficient coding of a device-specific haptic 439 effect. 441 Published specification: HAPT is a logical extension of the RIFF 442 standard [RIFF]. 444 Applications that use this media type: All applications that are able 445 to create, edit, or display haptic media content. 447 RFC XXXX The 'haptics' Top-level Media Type November 2021 449 Additional information: 451 * File extension(s): Haptic file extensions used for HAPT files: 452 .hapt 454 * Macintosh file type code(s): (no code specified) 456 * Macintosh Universal Type Identifier code: None 458 * Fragment Identifier: None 460 * Deprecated Alias: None 462 Person & email address to contact for further information: Yeshwant 463 Muthusamy(ymuthusamy@immersion.com) 465 Change controller: Immersion Corporation 467 5. Normative References 469 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 470 Requirement Levels", BCP 14, RFC 2119, 471 DOI 10.17487/RFC2119, March 1997, 472 . 474 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 475 Specifications and Registration Procedures", BCP 13, 476 RFC 6838, DOI 10.17487/RFC6838, January 2013, 477 . 479 6. Informative References 481 [ISOBMFF-IS] 482 "ISO/IEC 14496-12 (7th Edition) Information technology — 483 Coding of audio-visual objects — Part 12: ISO base media 484 file format", . 486 [MPEG-Haptics-Encoder] 487 "Encoder Input Format for MPEG Haptics", 488 . 491 [AHAP] "Apple Haptic Audio Pattern", 492 . 495 RFC XXXX The 'haptics' Top-level Media Type November 2021 497 [MPEG-Haptics-CfP] 498 "MPEG Evaluates Responses to the Haptics Phase 1 Call for 499 Proposals", 500 . 502 [W3C-Vibration] 503 "W3C Vibration API (Second Edition)", 504 . 506 [W3C-Gamepad] 507 "W3C Gamepad Extensions", 508 . 510 [IEEE-P191811] 511 "P1918.1.1 - Haptic Codecs for the Tactile Internet", 512 . 514 [Media-Type-Registration] 515 "IANA, Application for a Media Type", 516 . 518 [RIFF] "Resource Interchange File Format", 519 . 522 Authors' Addresses 524 Yeshwant K. Muthusamy 525 Immersion Corporation 526 330 Townsend St. Suite 234 527 San Francisco, CA 94107 528 United States of America 530 Phone: +1 469-583-2171 531 Email: ymuthusamy@immersion.com 533 Chris Ullrich 534 Immersion Corporation 535 330 Townsend St. Suite 234 536 San Francisco, CA 94107 537 United States of America 539 Phone: +1 805-320-0774 540 Email: cullrich@immersion.com