idnits 2.17.00 (12 Aug 2021) /tmp/idnits37692/draft-km-virt-orchestra-research-challenges-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (12 July 2021) is 306 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Independent Submission K. Makhijani 3 Internet-Draft L. Dong 4 Intended status: Informational Futurewei 5 Expires: 13 January 2022 12 July 2021 7 Virtual Orchestra Usecase and Research Challenges 8 draft-km-virt-orchestra-research-challenges-00 10 Abstract 12 This document describes open research challenges for emerging media- 13 oriented ensemble applications. One such driving scenario is the 14 network delivery of virtual orchestra that imposes multi-disciplinary 15 challenges. Specifically, of interest are the group communication 16 patterns in the production, delivery and consumption as different 17 dimensions relating to the communication networks. 18 This document brings forth current research and engineering 19 challenges in immersive media ensembles. The network domain problems 20 come down to the specification of coordination of the received 21 content with dependency constraints. The challenges depict both real 22 and quasi- realtime behavior. A number of endpoint actors get 23 involved in delivering the ensemble aspect, the research challenges 24 also describe the expectations from the end points. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on 13 January 2022. 43 Copyright Notice 45 Copyright (c) 2021 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 50 license-info) in effect on the date of publication of this document. 51 Please review these documents carefully, as they describe your rights 52 and restrictions with respect to this document. Code Components 53 extracted from this document must include Simplified BSD License text 54 as described in Section 4.e of the Trust Legal Provisions and are 55 provided without warranty as described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction and Scope . . . . . . . . . . . . . . . . . . . 2 60 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 3. Scenario Description . . . . . . . . . . . . . . . . . . . . 3 62 3.1. Multiple Streams and actors . . . . . . . . . . . . . . . 4 63 3.1.1. Conductor to Musicians . . . . . . . . . . . . . . . 4 64 3.1.2. Musicians to Stage . . . . . . . . . . . . . . . . . 5 65 3.2. Virtual Orchestra Scenario Challenges . . . . . . . . . . 5 66 4. Generlized Coordinated Service Concept . . . . . . . . . . . 6 67 5. Virtual Orchestra Coordination Challenges . . . . . . . . . . 7 68 5.1. Out-of-band Coordination . . . . . . . . . . . . . . . . 7 69 5.2. In-band Coordination . . . . . . . . . . . . . . . . . . 8 70 5.3. In-node Coordinated-forwarding . . . . . . . . . . . . . 8 71 6. Existing Research Work . . . . . . . . . . . . . . . . . . . 8 72 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 73 8. Security Considerations . . . . . . . . . . . . . . . . . . . 8 74 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8 76 1. Introduction and Scope 78 The multimedia segment has seen tremendous advancements in immersive 79 multimedia technologies. One of the ongoing research question is how 80 to deliver a complete immersion experience of the digital media. 81 Such media is produced from an ensemble of different actors or 82 multiple sources that must coordinate as they perform together in the 83 real environment. It translates to generating a very high volume of 84 generated data streams 86 This memo presents research and engineering challenges in multi-user 87 digital ensemble that need to be addressed in order to achieve these 88 goals, spanning from pure research and engineering/standards space. 89 The network related challenges are generalized as coordinated 90 communications and explained as group communications with explicit 91 dependencies. The objective of this memo is to document the 92 technical challenges and corresponding current approaches and to 93 expose requirements that should be addressed by future research and 94 standards work. 96 2. Terminology 98 * Co-flows: Co dependent flows 100 * Coordinated service: A network capability to support 101 communications between co-dependent flows 103 3. Scenario Description 105 In an orchestra ensemble the multimedia streams of musicians, each in 106 a different place in the world come together and perform live on the 107 stage which may also be at a different location. 109 * A conductor directs the sound of the ensemble with his gestures. 110 These gestures must be received at the same time by the remote 111 musicians at different locations to play their instruments at a 112 specific time with a specified tempo. 114 * Similarly, the music transmitted from those musicians' locations 115 to the stage must be played together with the same beats and 116 tempo. Any delay or early arrival of the sound from any one 117 instrument can cause the ensemble to go out of tune and destroy 118 the entire performance. 120 Performing an ensemble with multiple participants separated by large 121 as well as varying distances (from less than a mile, to 1000 miles) 122 is quite difficult for applications due to varying path and latency 123 characteristics. 125 The network needs to support the coordination of directions from the 126 conductor to all of the musicians and the audio/visuals from 127 musicians to the stage. In particular, in a large-scale ensemble 128 when many instruments are involved, in order to to preserve the 129 integrity of performance, it may be necessary to allow for the 130 dropping of sound and hologram streams of a musician that cannot 131 arrive at the same time as the others and to provide mechanisms for 132 subsequent fast synchronization. 134 Virtual orchestra is a complex multi-disciplinary use case and 135 requires in-depth knowledge in every field to recreate the real 136 orchestral experience. 138 +-------------+ 139 | conductor | 140 +-------------+ 141 | 142 | one to many 143 +--------------+--------------+ 144 | | | 145 v v v 146 +-----+ +-------+ +-------+ 147 | t0" | | t0 | | t0' | 148 +-(A)-+ +--(B)--+ +--(C)--+ 149 | | | 150 +-----------+ | +-----------+ 151 | | | many to one 152 v v v 153 +-----------------+ 154 | coordinator node| 155 +--------+--------+ 156 | 157 v 158 +--------+ 159 | Stage | 160 +--------+ 162 Figure 1: Virtual Orchestra Delivery over Network 164 3.1. Multiple Streams and actors 166 A virtual orchestra is a coordination of multiple flows as shown in 167 Figure 1. In the current network terminology this is equivalent to 168 multicast group of a number of endpoints and requiring to meet 169 cooperation between the endpoints on how to send and receive 170 information. An application point of view sees this as a membership 171 to publish/subscribe topic. In the above example, endpoint actors 172 are the conductor, musicians and the stage. The characteristics of 173 traffic are predictable and the following steps take place 175 3.1.1. Conductor to Musicians 177 * The conductor is initiator of the orchestral stream. Synchronized 178 reception of the gestures of a conductor are critical to the 179 performance. 181 * Musicians perform on cues or gestures received over the network. 182 It is necessary that all the musicians receive those cues to start 183 the performance. 185 * The performance follows the tempo and beats from the conductor, 186 which must be delivered in a consistent (jitter free) manner 187 (incurring no jitter). 189 3.1.2. Musicians to Stage 191 Atleast one output stream per sources will be generated to create the 192 ensemble performance, these sources may have variable latencies. 193 They should be aggregated to be delivered to the stage as a unified 194 stream. 196 The two scenarios are one to many Section 3.1.1 and many to 197 oneSection 3.1.2 type of group communication. The coordination 198 constraints involve several dependencies such as of synchronization 199 at the start of play, maintenance of same tempo along the time scale 200 throughout the streaming part, description of distance for spatial 201 sound quality. 203 3.2. Virtual Orchestra Scenario Challenges 205 In this section we draw forth scenarios with difficulties in 206 delivering virtual orchestra over the network. 208 Note that virtual orchestra application itself maybe delivered in 209 different ways. Non-realtime scenarios are not relevant since, in 210 that case it is a non-interactive content delivery, the content does 211 not require aggregation from multiple sources. An application and 212 corresponding network can use buffering, low latency techniques and 213 existing transport protocols to meet the expectations of an end-user. 215 Specific to real-time streaming of virtual orchestra, the performance 216 is pseudo-real-time. It means that the synchronization of content 217 originating from different sources is only as fast as its slowest 218 path. In other words, one source-destination path of the co-flow 219 will cause the pace of the group stream to slow-down, even though the 220 other, shorter latency paths may deliver content sooner. This in a 221 major co-dependency challenge, since the slowest path should not have 222 any impact on the tempo and the beats. Thus 3 dependency 223 considerations for the network are: - Feasibility Dependency: Assess 224 and determine that with the slowest flow-member of the group if such 225 a flow is even feasible. - Membership Dependency (spatial): The 226 mechanisms to establish and determine membership and establish 227 relationship is needed. Corelating to publisher (conductor and 228 stage) and subscriber (performers) group communication model, not all 229 subscribers need to know about each other. - Start Time Dependency 230 (temporal): Each performer depends on the trigger to start from there 231 on time-scale, tempo and beats of the performance must be preserved. 233 From a logical architectural point of view, coordination node is a 234 function that synchronizes all the incoming streams, it may then 235 either deliver all the streams or as a single stream. 237 4. Generlized Coordinated Service Concept 239 There are several examples of multi-party immersive applications (TBD 240 - add section) in which remote entities will be required to recreate 241 the behavior of being present in the same scene or environment. 242 Therefore, they are co-dependent on each other's spatio-temporal 243 behavior changes. For example, in an orchestra tempo or beats and 244 gestures must remain the same for all performers and position of a 245 musician is computed to create spatial audio. 247 A generalized in-network capability is introduced that consumes group 248 communication membership and constraints and delivers service with in 249 the specified constriaints. 251 Keeping in the network context, important terms and components of 252 coordinated service are introduced as below: 254 .-. 255 (---)---> 256 `-'----> 257 .---. +-----+ 258 ( )----------|Co-EP| 259 member `---' +-----+ 260 +-----+ | flow Co-SN 261 |Co-EP|----+ | ^^ 262 +-----+ | | .||. 263 | | ( ||) .-. 264 | v `||' ----(--)----> 265 .---. --------> .---. -----`-'-----> +-----+ 266 ( )-------------( )-------------------|Co-EP| 267 `---' -------> `---' +-----+ 268 +-----+ | Co-SN 269 |Co-EP|-----+ 270 +-----+ ------> Co-EP: coordinated service end point 271 Co-SN: coordinated service node 272 Co-Flow: coordinated flow 273 Member flow: member of a co-flow 275 * Coordinated Service: A coordinated service provides guarantee of 276 delivery of multiple flows in a dependent manner. It is a type of 277 group communication service supported by the network in which each 278 member of the group has a dependency on other group-members. A 279 coordinated service should be able to coordinate delivery of co- 280 flows over different categories of group communications. 282 * Coordinated Service end point (Co-EP): An endpoint in a group 283 communication where in coordination of resources or constraints is 284 required. 286 * Coordinated service Points (Co-SN): The mechanisms to support 287 coordinated services in network requires new capabilities referred 288 to as network coordination functions. A service node is a part of 289 the network that understands and processes network specific 290 functions of coordination. 292 * Co-dependent Flow(Co-Flow): A set of flows that have dependencies 293 in relation to eachother. Each flow in the co-dependent flow set 294 is referred to as a member-flow. The co-flows may express 295 different kinds of dependencies or relationships. It may be point 296 to mltipoint, multipoint to point or multipoint to multipoint. 298 * Member flow: A single point to point member flow of a co-flow. 300 Coordinated services are a form of group comminication with a clearly 301 expressed dependencies. Possible approaches will figure out 302 mechanisms to manage those dependencies. 304 5. Virtual Orchestra Coordination Challenges 306 The internet is a spatial-temporal heterogeneous environment, 307 yielding different content delivery behaviours in time and space. No 308 two paths (or even different flows on the same path) can be assumed 309 to have identical properties in terms of latency, jitter, and 310 bandwidth. 312 Currently, any effort to support virtual orchestra in the networks is 313 not feasible. Managing flow dependencis entirely by the applications 314 on endpoints does not always guarantee absolute time constraints due 315 to unpredictable changes in network conditions. This necessitates 316 some kind of coordination with the network. 318 5.1. Out-of-band Coordination 320 The out-of-band coordination may be used to achieve distribution of 321 coflows in the network. The membership of co-dependent flows is 322 conveyed from the end-points potentially when the flows are set up, 323 so that the coarse-grained service (and service level objectives) can 324 be enabled in the network. A distribution graph of coflows and 325 associated dependency constraints may be constructed, and those nodes 326 enhance their scheduling and forwarding by factoring in the timing 327 information in the meta-data of packets. 329 5.2. In-band Coordination 331 Then timestamping of transmission from sender delivery to receiver 332 may be conveyed as meta-data in packets transmitted from the senders. 333 This type of In-band signaling conveys intermediate coordination 334 points about the dependencies and interrelationship. To formalize 335 these mechanisms to carry them in data path. 337 5.3. In-node Coordinated-forwarding 339 Actual coordination effort is done on the coordination points. The 340 scheduling and forwarding engine should allow packets within sync 341 markers to be sent as per remaining δt. It needs to compare the 342 remaining coordination time and accordingly schedule or pace the 343 packet forwarding. 345 6. Existing Research Work 347 7. IANA Considerations 349 This document requires no actions from IANA. 351 8. Security Considerations 353 This document introduces no new security issues. 355 Authors' Addresses 357 Kiran Makhijani 358 Futurewei 360 Email: kiran.ietf@gmail.com 362 Lijun Dong 363 Futurewei 364 Central Expy 365 Santa Clara, CA 95050, 366 United States of America 368 Email: lijun.dong@futurewei.com