idnits 2.17.00 (12 Aug 2021) /tmp/idnits38904/draft-irtf-t2trg-iot-edge-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document date (11 January 2022) is 123 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-14) exists of draft-ietf-core-oscore-groupcomm-13 Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Hong 3 Internet-Draft ETRI 4 Intended status: Informational Y-G. Hong 5 Expires: 15 July 2022 Daejeon University 6 X. de Foy 7 InterDigital Communications, LLC 8 M. Kovatsch 9 Huawei Technologies Duesseldorf GmbH 10 E. Schooler 11 Intel 12 D. Kutscher 13 University of Applied Sciences Emden/Leer 14 11 January 2022 16 IoT Edge Challenges and Functions 17 draft-irtf-t2trg-iot-edge-04 19 Abstract 21 Many IoT applications have requirements that cannot be met by the 22 traditional Cloud (aka cloud computing). These include time 23 sensitivity, data volume, connectivity cost, operation in the face of 24 intermittent services, privacy, and security. As a result, the IoT 25 is driving the Internet toward Edge computing. This document 26 outlines the requirements of the emerging IoT Edge and its 27 challenges. It presents a general model, and major components of the 28 IoT Edge, to provide a common base for future discussions in T2TRG 29 and other IRTF and IETF groups. 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on 15 July 2022. 48 Copyright Notice 50 Copyright (c) 2022 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 55 license-info) in effect on the date of publication of this document. 56 Please review these documents carefully, as they describe your rights 57 and restrictions with respect to this document. Code Components 58 extracted from this document must include Revised BSD License text as 59 described in Section 4.e of the Trust Legal Provisions and are 60 provided without warranty as described in the Revised BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 3 66 2.1. Internet of Things (IoT) . . . . . . . . . . . . . . . . 3 67 2.2. Cloud Computing . . . . . . . . . . . . . . . . . . . . . 4 68 2.3. Edge Computing . . . . . . . . . . . . . . . . . . . . . 4 69 2.4. Examples of IoT Edge Computing Use Cases . . . . . . . . 6 70 3. IoT Challenges Leading Towards Edge Computing . . . . . . . . 9 71 3.1. Time Sensitivity . . . . . . . . . . . . . . . . . . . . 9 72 3.2. Connectivity Cost . . . . . . . . . . . . . . . . . . . . 10 73 3.3. Resilience to Intermittent Services . . . . . . . . . . . 10 74 3.4. Privacy and Security . . . . . . . . . . . . . . . . . . 10 75 4. IoT Edge Computing Functions . . . . . . . . . . . . . . . . 11 76 4.1. Overview of IoT Edge Computing Today . . . . . . . . . . 11 77 4.2. General Model . . . . . . . . . . . . . . . . . . . . . . 13 78 4.3. OAM Components . . . . . . . . . . . . . . . . . . . . . 16 79 4.3.1. Resource Discovery and Authentication . . . . . . . . 16 80 4.3.2. Edge Organization and Federation . . . . . . . . . . 17 81 4.3.3. Multi-Tenancy and Isolation . . . . . . . . . . . . . 18 82 4.4. Functional Components . . . . . . . . . . . . . . . . . . 18 83 4.4.1. In-Network Computation . . . . . . . . . . . . . . . 18 84 4.4.2. Edge Storage and Caching . . . . . . . . . . . . . . 20 85 4.4.3. Northbound/Southbound Communication . . . . . . . . . 20 86 4.4.4. Communication Brokering . . . . . . . . . . . . . . . 21 87 4.5. Application Components . . . . . . . . . . . . . . . . . 21 88 4.5.1. IoT End Devices Management . . . . . . . . . . . . . 21 89 4.5.2. Data Management and Analytics . . . . . . . . . . . . 22 90 4.6. Simulation and Emulation Environments . . . . . . . . . . 23 91 5. Security Considerations . . . . . . . . . . . . . . . . . . . 23 92 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 24 93 7. Informative References . . . . . . . . . . . . . . . . . . . 24 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 96 1. Introduction 98 Currently, many IoT services leverage the Cloud, since it can provide 99 virtually unlimited storage and processing power. The reliance of 100 IoT on back-end cloud computing brings additional advantages such as 101 flexibility and efficiency. Today's IoT systems are fairly static 102 with respect to integrating and supporting computation. It's not 103 that there is no computation, but systems are often limited to static 104 configurations (edge gateways, cloud services). 106 However, IoT devices are creating vast amounts of data at the network 107 edge. To meet IoT use case requirements, that data increasingly is 108 being stored, processed, analyzed, and acted upon close to the data 109 producers. These requirements include time sensitivity, data volume, 110 connectivity cost, resiliency in the face of intermittent 111 connectivity, privacy, and security, which cannot be addressed by 112 today's centralized cloud computing. These requirements suggest a 113 more flexible way to distribute computing (and storage) and to 114 integrate it in the edge-cloud continuum. We will refer to this 115 integration of edge computing and IoT as "IoT edge computing". Our 116 draft describes related background, uses cases, challenges, system 117 models, and functional components. 119 Due to the dynamic nature of the IoT edge computing landscape, this 120 document does not list existing projects in this field. However, 121 Section 4.1 presents a high-level overview of the field, based on a 122 limited review of standards, research, open-source and proprietary 123 products in [I-D-defoy-t2trg-iot-edge-computing-background]. 125 2. Background 127 2.1. Internet of Things (IoT) 129 Since the term "Internet of Things" (IoT) was coined by Kevin Ashton 130 in 1999 working on Radio-Frequency Identification (RFID) technology 131 [Ashton], the concept of IoT has evolved. It now reflects a vision 132 of connecting the physical world to the virtual world of computers 133 using (wireless) networks over which things can send and receive 134 information without human intervention. Recently, the term has 135 become more literal by actually connecting things to the Internet and 136 converging on Internet and Web technology. 138 Things are usually embedded systems of various kinds, such as home 139 appliances, mobile equipment, wearable devices, etc. Things are 140 widely distributed, but typically have limited storage and processing 141 power, which raise concerns regarding reliability, performance, 142 energy consumption, security, and privacy [Lin]. This limited 143 storage and processing power leads to complementing IoT with cloud 144 computing. 146 2.2. Cloud Computing 148 Cloud computing has been defined in [NIST]: "cloud computing is a 149 model for enabling ubiquitous, convenient, on-demand network access 150 to a shared pool of configurable computing resources (e.g., networks, 151 servers, storage, applications, and services) that can be rapidly 152 provisioned and released with minimal management effort or service 153 provider interaction". Low cost and massive availability of storage 154 and processing power enabled the realization of another computing 155 model, in which virtualized resources can be leased in an on-demand 156 fashion, being provided as general utilities. Companies like Amazon, 157 Google, Facebook, etc. widely adopted this paradigm for delivering 158 services over the Internet, gaining both economical and technical 159 benefits [Botta]. 161 Today, an unprecedented volume and variety of data is generated by 162 things, and applications deployed at the network edge consume this 163 data. In this context, cloud-based service models are not suitable 164 for some classes of applications, which for example need very short 165 response times, access to local personal data, or generate vast 166 amounts of data. Those applications may instead leverage edge 167 computing. 169 2.3. Edge Computing 171 Edge computing, also referred to as fog computing in some settings, 172 is a new paradigm in which substantial computing and storage 173 resources are placed at the edge of the Internet, that is, close to 174 mobile devices, sensors, actuators, or machines. Edge computing 175 happens near data sources [Mahadev], or closer (topologically, 176 physically, in terms of latency, etc.) to where decisions or 177 interactions with the physical world are happening. It processes 178 both downstream data, e.g. originated from cloud services, and 179 upstream data, e.g. originated from end devices or network elements. 180 The term fog computing usually represents the notion of a multi- 181 tiered edge computing, that is, several layers of compute 182 infrastructure between the end devices and cloud services. 184 An edge device is any computing or networking resource residing 185 between end-devices' data sources and cloud-based data centers. In 186 edge computing, end devices not only consume data but also produce 187 data. And at the network edge, devices not only request services and 188 information from the Cloud, but also handle computing tasks including 189 processing, storage, caching, and load balancing on data sent to and 190 from the Cloud [Shi]. This does not preclude end devices from 191 hosting computation themselves when possible, independently or as 192 part of a distributed edge computing platform (this is also referred 193 to as Mist Computing). 195 Several standards developing organization (SDO) and industry forums 196 have provided definitions of edge and fog computing: 198 * ISO defines edge computing as a "form of distributed computing in 199 which significant processing and data storage takes place on nodes 200 which are at the edge of the network" [ISO_TR]. 202 * ETSI defines multi-access edge computing as a "system which 203 provides an IT service environment and cloud-computing 204 capabilities at the edge of an access network which contains one 205 or more type of access technology, and in close proximity to its 206 users" [ETSI_MEC_01]. 208 * The Industrial IoT Consortium (IIC, formerly OpenFog) defines fog 209 computing as "a horizontal, system-level architecture that 210 distributes computing, storage, control and networking functions 211 closer to the users along a cloud-to-thing continuum" [OpenFog]. 213 Based on these definitions, we can summarize a general philosophy of 214 edge computing as to distribute the required functions close to users 215 and data, while the difference to classic local systems is the usage 216 of management and orchestration features adopted from cloud 217 computing. 219 Actors from various industries approach edge computing using 220 different terms and reference models, although in practice these 221 approaches are not incompatible and may integrate with each other: 223 * The telecommunication industry tends to use a model where edge 224 computing services are deployed over Network Function 225 Virtualization (NFV) infrastructure, at aggregation points or in 226 proximity to the user equipment (e.g., gNodeBs) [ETSI_MEC_03]. 228 * Enterprise and campus solutions often interpret edge computing as 229 an "edge cloud", that is, a smaller data center directly connected 230 to the local network (often referred to as "on-premise"). 232 * The automation industry defines the edge as the connection point 233 between IT from OT (Operational Technology). Hence, here edge 234 computing sometimes refers to applying IT solutions to OT problems 235 such as analytics, more flexible user interfaces, or simply having 236 more computing power than an automation controller. 238 2.4. Examples of IoT Edge Computing Use Cases 240 IoT edge computing can be used in home, industry, grid, healthcare, 241 city, transportation, agriculture, and/or education scenarios. We 242 discuss here only a few examples of such use cases, to point out 243 differentiating requirements. These examples are followed with 244 references to other use cases. 246 *Smart Factory* 248 As part of the 4th industrial revolution, smart factories run real- 249 time processes based on IT technologies such as artificial 250 intelligence and big data. In a smart factory, even a very small 251 environmental change can lead to a situation in which production 252 efficiency decreases or product quality problems occur. Therefore, 253 simple but time-sensitive processing can be performed at the edge: 254 for example, controlling temperature and humidity in the factory, or 255 operating machines based on the real-time collection of the 256 operational status of each machine. On the other hand, data 257 requiring highly precise analysis, such as machine lifecycle 258 management or accident risk prediction, can be transferred to a 259 central data center for processing. 261 The use of edge computing in a smart factory can reduce the cost of 262 network and storage resources by reducing the communication load to 263 the central data center or server. It is also possible to improve 264 process efficiency and facility asset productivity through the real- 265 time prediction of failures, and to reduce the cost of failure 266 through preliminary measures. In the existing manufacturing field, 267 production facilities are manually run according to a program entered 268 in advance, but edge computing in a smart factory enables tailoring 269 solutions by analyzing data at each production facility and machine 270 level. Digital twins [Jones] of IoT devices have been use jointly 271 with edge computing in industrial IoT scenarios [Chen]. 273 *Smart Grid* 275 In future smart city scenarios, the Smart Grid will be critical in 276 ensuring highly available/efficient energy control in city-wide 277 electricity management. Edge computing is expected to play a 278 significant role in those systems to improve transmission efficiency 279 of electricity; to react and restore power after a disturbance; to 280 reduce operation costs and reuse renewable energy effectively, since 281 these operations involve local decision-making. In addition, edge 282 computing can help to monitor power generation and power demand, and 283 making local electrical energy storage decisions in the smart grid 284 system. 286 *Smart Agriculture* 288 Smart agriculture integrates information and communication technology 289 with farming technology. Intelligent farms use IoT technology to 290 measure and analyze temperature, humidity, sunlight, carbon dioxide, 291 soil, etc. in crop cultivation facilities. Depending on analysis 292 results, control devices are used to set environmental parameters to 293 an appropriate state. Remote management is also possible through 294 mobile devices such as smartphones. 296 In existing farms, simple systems such as management according to 297 temperature and humidity can easily and inexpensively be implemented 298 with IoT technology. Sensors in fields are gathering data on field 299 and crop condition. This data is then transmitted to cloud servers, 300 which process data and recommend actions. Usage of edge computing 301 can reduce by a large amount data transmitted up and down the 302 network, resulting in saving cost and bandwidth. Locally generated 303 data can be processed at the edge, and local computing and analytics 304 can drive local actions. With edge computing, it is also easy for 305 farmers to select large amounts of data for processing, and data can 306 be analyzed even in remote areas with poor access conditions. As the 307 number of people working on farming decreases over time, increasing 308 automation enabled by edge computing can be a driving force for 309 future smart agriculture. 311 *Smart Construction* 313 Safety is critical on a construction site. Every year, many 314 construction workers lose their lives due to falls, collisions, 315 electric shocks, and other accidents. Therefore, solutions have been 316 developed in order to improve construction site safety, including 317 real-time identification of workers, monitoring of equipment 318 location, and predictive accident prevention. To deploy these 319 solutions, many cameras and IoT sensors were installed on 320 construction sites, that measure noise, vibration, gas concentration, 321 etc. Typically, data generated from these measurements has been 322 collected in an on-site gateway and sent to a remote cloud server for 323 storage and analysis. Thus, an inspector can check the information 324 stored on the cloud server to investigate an incident. However, this 325 approach can be expensive, due to transmission costs, e.g., of video 326 streams over an LTE connection, and due to usage fees of private 327 cloud services such as Amazon Web Services. 329 Using edge computing, data generated on the construction site can be 330 processed and analyzed on an edge server located within or near the 331 site. Only the result of this processing needs to be transferred to 332 a cloud server, thus saving transmission costs. It is also possible 333 to locally generate warnings to prevent accident in real-time. 335 *Self-Driving Car* 337 The self-driving car, with its focus on safety, is a system where 338 edge computing has an essential role. Autonomous vehicles are 339 equipped with high-resolution cameras, radars, laser scanners 340 (LIDAR), sonar sensors, and GPS systems. Edge computing nodes 341 collect and analyze vast amounts of data generated in real-time by 342 these sensors to keep track of distances between vehicles in front, 343 surrounding road conditions, vehicle flow, and to quickly respond to 344 unexpected situations. For example, if the speed of the car running 345 in front decreases, speed should be adjusted to maintain the distance 346 between the cars, and when a roadside signal changes, a self-driving 347 car should operate according to the new signal. If such processing 348 is performed in a central data center, network delays or data 349 transmission errors can lead to accidents. Applying edge computing 350 can minimize these network delays and data transmission errors, 351 thereby improving safety. In the shorter term we can expect edge 352 computing nodes to be at the base station or in road-side units. 353 However, to further reduce reaction times, some edge computing nodes 354 should be located in the vehicle itself. 356 *AR/VR* 358 Augmented Reality (AR) and Virtual Reality (VR) are likely to 359 strongly influence the Information and Communication Technology (ICT) 360 market in the future, since they can support innovative products in 361 most other use cases including smart factories, self-driving cars, 362 etc. In AR/VR, due to large amounts of data generated at endpoints 363 such as mobile devices and PCs, user immersion can be significantly 364 decreased by a latency of only a few hundred milliseconds. 365 Therefore, using an edge computing infrastructure built close to 366 endpoints can not only reduce the cost and latency of data 367 transmission but also maximize user immersion. For example, in AR 368 using edge computing, streaming video can be displayed realistically 369 in higher quality, giving users the best possible experience. 371 *Other Use Cases* 373 Additionally, oneM2M recently studied several use cases related to 374 edge computing, including: smart factories, smart transportation, an 375 accident notification service, a high-precision road map service, a 376 vulnerable road user service and a vehicular data service. These use 377 cases are documented in [oneM2M-TR0001], [oneM2M-TR0018] and 378 [oneM2M-TR0026]. Edge computing related requirements raised through 379 the analysis of these use cases are captured in [oneM2M-TS0002]. 381 3. IoT Challenges Leading Towards Edge Computing 383 This section describes challenges met by IoT, that are motivating the 384 adoption of edge computing for IoT. Those are distinct from research 385 challenges applicable to IoT edge computing, some of which will be 386 mentioned in Section 4.3. 388 IoT technology is used with more and more demanding applications, 389 e.g. in industrial, automotive or healthcare domains, leading to new 390 challenges. For example, industrial machines such as laser cutters 391 already produce over 1 terabyte per hour, and similar amounts can be 392 generated in autonomous cars [NVIDIA]. 90% of IoT data is expected to 393 be stored, processed, analyzed, and acted upon close to the source 394 [Kelly], as cloud computing models alone cannot address the new 395 challenges [Chiang]. 397 Below we discuss IoT use case requirements that are moving cloud 398 capabilities to be more proximate and more distributed and 399 disaggregated. 401 3.1. Time Sensitivity 403 Many industrial control systems, such as manufacturing systems, smart 404 grids, oil and gas systems, etc., often require stringent end-to-end 405 latency between the sensor and control node. While some IoT 406 applications may require latency below a few tens of milliseconds 407 [Weiner], industrial robots and motion control systems have use cases 408 for cycle times in the order of microseconds [_60802]. In some cases 409 speed-of-light limitations may simply prevent a solution based on 410 remote cloud, however it is not the only challenge relative to time 411 sensitivity. Guarantees for jitter are also important to those 412 industrial IoT applications. This means control packets need to 413 arrive with as little variation as possible and within a strict 414 deadline. Given the best-effort characteristics of the Internet, 415 this challenge is virtually impossible to address, without using end- 416 to-end guarantees for individual message delivery and continuous data 417 flows. 419 3.2. Connectivity Cost 421 Some IoT deployments are not challenged by a constrained network 422 bandwidth to the Cloud. The fifth generation mobile networks (5G) 423 and Wi-Fi 6 both theoretically top out at 10 gigabits per second 424 (i.e., 4.5 terabytes per hour), which enables high-bandwidth uplinks. 425 However, the resulting cost for high-bandwidth connectivity to upload 426 all data to the Cloud is unjustifiable and impractical for most IoT 427 applications. In some settings, e.g. in aeronautical communication, 428 higher communication costs reduce the amount of data that can be 429 practically uploaded even further. 431 3.3. Resilience to Intermittent Services 433 Many IoT devices such as sensors, data collectors, actuators, 434 controllers, etc. have very limited hardware resources and cannot 435 rely solely on their limited resources to meet all their computing 436 and/or storage needs. They require reliable, uninterrupted, or 437 resilient services to augment their capabilities in order to fulfill 438 their application tasks. This is hard and partly impossible to 439 achieve with cloud services for systems such as vehicles, drones, or 440 oil rigs that have intermittent network connectivity. The dual is 441 also true, a cloud back-end might want to have a reading of the 442 device even if it's currently asleep. 444 3.4. Privacy and Security 446 When IoT services are deployed at home, personal information can be 447 learned from detected usage data. For example, one can extract 448 information about employment, family status, age, and income by 449 analyzing smart meter data [ENERGY]. Policy-makers started to 450 provide frameworks that limit the usage of personal data and put 451 strict requirements on data controllers and processors. However, 452 data stored indefinitely in the Cloud also increases the risk of data 453 leakage, for instance, through attacks on rich targets. 455 Industrial systems are often argued to not have privacy implications, 456 as no personal data is gathered. Yet data from such systems is often 457 highly sensitive, as one might be able to infer trade secrets such as 458 the setup of production lines. Hence, the owners of these systems 459 are generally reluctant to upload IoT data to the Cloud. 461 Furthermore, passive observers can perform traffic analysis on the 462 device-to-cloud path. Hiding traffic patterns associated with sensor 463 networks can therefore be another requirement for edge computing. 465 4. IoT Edge Computing Functions 467 In this section, we first look at the current state of IoT edge 468 computing Section 4.1, and then define a general system model 469 Section 4.2. This provides context for IoT edge computing functions, 470 which are listed in Section 4.3. 472 4.1. Overview of IoT Edge Computing Today 474 This section provides an overview of today's IoT edge computing 475 field, based on a limited review of standards, research, open-source 476 and proprietary products in 477 [I-D-defoy-t2trg-iot-edge-computing-background]. 479 IoT gateways, both open-source (such as EdgeX Foundry or Home Edge) 480 and proprietary (such as Amazon Greengrass, Microsoft Azure IoT Edge, 481 Google Cloud IoT Core, and gateways from Bosh, Siemens), represent a 482 common class of IoT edge computing products, where the gateway is 483 providing a local service on customer premises and is remotely 484 managed through a cloud service. IoT communication protocols are 485 typically used between IoT devices and the gateway, including CoAP, 486 MQTT, and many specialized IoT protocols (such as OPC UA and DDS in 487 the Industrial IoT space), while the gateway communicates with the 488 distant cloud typically using HTTPS. Virtualization platforms enable 489 the deployment of virtual edge computing functions (using VMs, 490 application containers, etc.), including IoT gateway software, on 491 servers in the mobile network infrastructure (at base stations and 492 concentration points), in edge data centers (in central offices) or 493 regional data centers located near central offices. End devices are 494 envisioned to become computing devices in forward-looking projects, 495 but they are not commonly used as such today. 497 Besides open-source and proprietary solutions, a horizontal IoT 498 service layer is standardized by the oneM2M standards body, to reduce 499 fragmentation, increase interoperability and promote reuse in the IoT 500 ecosystem. 502 Physical or virtual IoT gateways can host application programs, which 503 are typically built using an SDK to access local services through a 504 programmatic API. Edge cloud system operators host their customers' 505 applications VMs or containers on servers located in or near access 506 networks, which can implement local edge services. For example, 507 mobile networks can provide edge services for radio network 508 information, location, and bandwidth management. 510 Life cycle management of services and applications on physical IoT 511 gateways is often cloud-based. Edge cloud management platforms and 512 products (such as StarlingX, Akraino Edge Stack, Mobile EdgeX) adapt 513 cloud management technologies (e.g., Kubernetes) to the edge cloud, 514 i.e., to smaller, distributed computing devices running outside a 515 controlled data center. Services and application life-cycle is 516 typically using an NFV-like management and orchestration model. 518 Resilience in IoT often entails the ability to operate autonomously 519 in periods of disconnectedness in order to preserve the integrity and 520 safety of the controlled system, possibly in a degraded mode. IoT 521 devices and gateways are often expected to operate in the always-on 522 and unattended mode, using fault detection and unassisted recovery 523 functions. 525 The platform typically includes services to advertise or consume APIs 526 (e.g., Mp1 interface in ETSI MEC supports service discovery and 527 communication), and enables communicating with local and remote 528 endpoints (e.g., message routing function in IoT gateways). The 529 service platform is typically extensible by edge applications, since 530 they can advertise an API that other edge applications can consume. 531 IoT communication services include protocols translation, analytics, 532 and transcoding. Communication between edge computing devices is 533 enabled in tiered deployments or distributed deployments. 535 An edge cloud platform may enable pass-through without storage or 536 local storage (e.g., on IoT gateways). Some edge cloud platforms use 537 a distributed form of storage such as an ICN network, e.g., Named 538 Function Networking (NFN) nodes can store data in a Named Data 539 Networking (NDN) system, or a distributed storage platform (e.g., 540 IPFS, EdgeFS, Ceph). External storage, e.g., on databases in distant 541 or local IT cloud, is typically used for filtered data deemed worthy 542 of long-term storage, although in some cases it may be for all data, 543 for example when required for regulatory reasons. 545 Stateful computing is supported on platforms hosting native programs, 546 VMs or containers. Stateless computing is supported on platforms 547 providing a "serverless computing" service (a.k.a. function-as- 548 a-service, e.g., using stateless containers), or on systems based on 549 named function networking. 551 In many IoT use cases, a typical network usage pattern is high volume 552 uplink with some form of traffic reduction enabled by processing over 553 edge computing devices. Alternatives to traffic reduction include 554 deferred transmission (to off-peak hours or using physical shipping). 555 Downlink traffic includes application control and software updates. 556 Other, downlink-heavy traffic patterns are not excluded but are more 557 often associated with non-IoT usage (e.g., video CDNs). 559 4.2. General Model 561 Edge computing is expected to play an important role in deploying new 562 IoT services integrated with Big Data and AI, enabled by flexible in- 563 network computing platforms. Although there are lots of approaches 564 to edge computing, we attempt to lay out a general model and list 565 associated logical functions in this section. In practice, this 566 model can map to different architectures, such as: 568 * A single IoT gateway, or a hierarchy of IoT gateways, typically 569 connected to the cloud (e.g., to extend the traditional cloud- 570 based management of IoT devices and data to the edge). A common 571 role of an IoT Gateway is to provide access to a heterogeneous set 572 of IoT devices/sensors; handle IoT data; and deliver IoT data to 573 its final destination in a cloud network. Whereas an IoT gateway 574 needs interactions with the cloud, it can also operate 575 independently in a disconnected mode. 577 * A set of distributed computing nodes, e.g., embedded in switches, 578 routers, edge cloud servers, or mobile devices. Some IoT end 579 devices can have enough computing capabilities to participate in 580 such distributed systems due to advances in hardware technology. 581 In this model, edge computing nodes can collaborate to share their 582 resources. 584 In the general model described in Figure 1, the edge computing domain 585 is interconnected with IoT end devices (southbound connectivity) and 586 possibly with a remote/cloud network (northbound connectivity), and 587 with a service operator's system. Edge computing nodes provide 588 multiple logical functions, or components, which may not all be 589 present in a given system. They may be implemented in a centralized 590 or distributed fashion, at the network edge, or through some 591 interworking between edge network and remote cloud network. 593 +---------------------+ 594 | Remote network | +---------------+ 595 |(e.g., cloud network)| | Service | 596 +-----------+---------+ | Operator | 597 | +------+--------+ 598 | | 599 +--------------+-------------------+-----------+ 600 | Edge Computing Domain | 601 | | 602 | One or more Computing Nodes | 603 | (IoT gateway, end devices, switches, | 604 | routers, mini/micro-data centers, etc.) | 605 | | 606 | OAM Components | 607 | - Resource Discovery and Authentication | 608 | - Edge Organization and Federation | 609 | - Multi-Tenancy and Isolation | 610 | - ... | 611 | | 612 | Functional Components | 613 | - In-Network Computation | 614 | - Edge Caching | 615 | - North/South-bound Communication | 616 | - Communication Brokering | 617 | - Other Services | 618 | - ... | 619 | | 620 | Application Components | 621 | - IoT End Devices Management | 622 | - Data Management and Analytics | 623 | - ... | 624 | | 625 +------+--------------+-------- - - - -+- - - -+ 626 | | | | | 627 | | +-----+--+ 628 +----+---+ +-----+--+ | |compute | | 629 | End | | End | ... |node/end| 630 |Device 1| |Device 2| ...| |device n| | 631 +--------+ +--------+ +--------+ 632 + - - - - - - - -+ 634 Figure 1: Model of IoT Edge Computing 636 In the distributed model described in Figure 2, the edge computing 637 domain is composed of IoT edge gateways and IoT end devices which are 638 also used as computing nodes. Edge computing domains are connected 639 with a remote/cloud network, and with their respective service 640 operator's system. IoT end devices/computing nodes provide logical 641 functions, as part of a distributed machine learning application. 642 The processing capabilities in IoT end devices being limited, they 643 require the support of other nodes: the training process for AI 644 services is executed at IoT edge gateways or cloud networks and the 645 prediction (inference) service is executed in the IoT end devices. 647 +----------------------------------------------+ 648 | Edge Computing Domain | 649 | | 650 | +--------+ +--------+ +--------+ | 651 | |Compute | |Compute | |Compute | | 652 | |node/End| |node/End| .... |node/End| | 653 | |device 1| |device 2| .... |device m| | 654 | +----+---+ +----+---+ +----+---+ | 655 | | | | | 656 | +---+-------------+-----------------+--+ | 657 | | IoT Edge Gateway | | 658 | +-----------+-------------------+------+ | 659 | | | | 660 +--------------+-------------------+-----------+ 661 | | 662 +-----------+---------+ +------+-------+ 663 | Remote network | | Service | 664 |(e.g., cloud network)| | Operator(s) | 665 +-----------+---------+ +------+-------+ 666 | | 667 +--------------+-------------------+-----------+ 668 | | | | 669 | +-----------+-------------------+------+ | 670 | | IoT Edge Gateway | | 671 | +---+-------------+-----------------+--+ | 672 | | | | | 673 | +----+---+ +----+---+ +----+---+ | 674 | |Compute | |Compute | |Compute | | 675 | |node/End| |node/End| .... |node/End| | 676 | |device 1| |device 2| .... |device n| | 677 | +--------+ +--------+ +--------+ | 678 | | 679 | Edge Computing Domain | 680 +----------------------------------------------+ 682 Figure 2: Example: Machine Learning over a Distributed IoT Edge 683 Computing System 685 We now attempt to enumerate major edge computing domain components. 686 They are here loosely organized into OAM (Operations, Administration, 687 and Maintenance), functional and application components, with the 688 understanding that the distinction between these classes may not 689 always be clear, depending on actual system architectures. Some 690 representative research challenges are associated with those 691 functions. We used input from co-authors, IRTF attendees, and some 692 comprehensive reviews of the field ([Yousefpour], [Zhang2], [Khan]). 694 4.3. OAM Components 696 Edge computing OAM goes beyond the network-related OAM functions 697 listed in [RFC6291]. Besides infrastructure (network, storage, and 698 computing resources), edge computing systems can also include 699 computing environments (for VMs, software containers, functions), IoT 700 end devices, data, and code. 702 Operation-related functions include performance monitoring for 703 service level agreement measurement; fault management and 704 provisioning for links, nodes, compute and storage resources, 705 platforms, and services. Administration covers network/compute/ 706 storage resources, platforms and services discovery, configuration, 707 and planning. Discovery during normal operation (e.g., discovery of 708 compute nodes by endpoints) would typically not be included in OAM, 709 however in this document we will not address it separately. 710 Management covers monitoring and diagnostics of failures, as well as 711 means to minimize their occurrence and take corrective actions. This 712 may include software updates management, high service availability 713 through redundancy and multipath communication. Centralized (e.g., 714 SDN) and decentralized management systems can be used. Finally, we 715 arbitrarily chose to address data management as an application 716 component, however in some systems data management may be considered 717 to be similar to a network management function. 719 We further detail a few OAM components. 721 4.3.1. Resource Discovery and Authentication 723 Discovery and authentication may target platforms, infrastructure 724 resources, such as compute, network and storage, but also other 725 resources such as IoT end devices, sensors, data, code units, 726 services, applications, or users interacting with the system. 727 Broker-based solutions can be used, e.g., using an IoT gateway as a 728 broker to discover IoT resources. More decentralized solutions can 729 also be used in replacement or complement, e.g., CoAP enables 730 multicast discovery of an IoT device, and CoAP service discovery 731 enables obtaining a list of resources made available by this device 732 [RFC7252]. Today, centralized gateway-based systems rely, for device 733 authentication, on the installation of a secret on IoT end devices 734 and computing devices (e.g., a device certificate stored in a 735 hardware security module, or a combination of code and data stored in 736 a trusted execution environment). 738 Related challenges include: 740 * Discovery, authentication, and trust establishment between end 741 devices, compute nodes, and platforms, with regard to concerns 742 such as mobility, heterogeneous devices and networks, scale, 743 multiple trust domains, constrained devices, anonymity, and 744 traceability 746 * Intermittent connectivity to the Internet, preventing relying on a 747 third-party authority [Echeverria] 749 * Resiliency to failures [Harchol], denial of service attacks, 750 easier physical access for attackers 752 4.3.2. Edge Organization and Federation 754 In a distributed system context, once edge devices have discovered 755 and authenticated each other, they can be organized, or self- 756 organize, into hierarchies or clusters. The organization structure 757 may range from centralized to peer-to-peer, or it may be closely tied 758 with other systems. Such groups can also form federations with other 759 edge or remote clouds. 761 Related challenges include: 763 * Support for scaling, and enabling fault-tolerance or self-healing 764 [Jeong]. Besides using hierarchical organization to cope with 765 scaling, another available and possibly complementary mechanism is 766 multicast ([RFC7390] [I-D.ietf-core-oscore-groupcomm]) 768 * Integration of edge computing with virtualized Radio Access 769 Networks (Fog RAN) [I-D.bernardos-sfc-fog-ran] and with 5G access 770 networks 772 * Sharing resources in multi-vendor/operator scenarios, to optimize 773 criteria such as profit [Anglano], resource usage, latency, or 774 energy consumption 776 * Capacity planning, placement of infrastructure nodes to minimize 777 delay [Fan], cost, energy, etc. 779 * Incentives for participation, e.g. in peer-to-peer federation 780 schemes 782 4.3.3. Multi-Tenancy and Isolation 784 Some IoT edge computing systems make use of virtualized (compute, 785 storage and networking) resources to address the need for secure 786 multi-tenancy at the edge. This leads to "edge clouds" that share 787 properties with the remote Cloud and can reuse some of its ecosystem. 788 Virtualization function management is covered to a large extent by 789 ETSI NFV and MEC standards activities. Projects such as [LFEDGE-EVE] 790 further cover virtualization and its management into distributed edge 791 computing settings. 793 Related challenges include: 795 * Adapting cloud management platforms to the edge, to account for 796 its distributed nature, e.g., using Conflict-free Replicated Data 797 Types (CRDT) [Jeffery], heterogeneity and customization, e.g., 798 using intent-based management mechanisms [Cao], and limited 799 resources. 801 * Minimizing virtual function instantiation time and resource usage 803 4.4. Functional Components 805 4.4.1. In-Network Computation 807 A core function of IoT edge computing is to enable local computation 808 on a node at the network edge, e.g. processing input data from 809 sensors, making local decisions, preprocessing data, offloading 810 computation on behalf of a device, service, or user. Related 811 functions include orchestrating computation (in a centralized or 812 distributed manner) and managing applications lifecycle. Support for 813 in-network computation may vary in term of capability, e.g., 814 computing nodes can host virtual machines, software containers, 815 software actors or unikernels able to run stateful or stateless code, 816 or a rules engine providing an API to register actions in response to 817 conditions such as IoT device ID, sensor values to check, thresholds, 818 etc. 820 Edge offloading include offloading to and from an IoT device, and to 821 and from a network node. [Cloudlets] offer an example of offloading 822 from an end device to a network node. On the other side, oneM2M is 823 an example of a system that allows a cloud-based IoT platform to 824 transfer resources and tasks to a target edge node [oneM2M-TR0052]. 825 Once transferred, the edge node can directly support IoT devices it 826 serves with the service offloaded by the cloud (e.g. group 827 management, location management, etc.) 828 QoS can be provided in some systems through the combination of 829 network QoS (e.g., traffic engineering or wireless resource 830 scheduling) and compute/storage resource allocations. For example, 831 in some systems, a bandwidth manager service can be exposed to enable 832 allocation of bandwidth to/from an edge computing application 833 instance. 835 In-network computation may leverage underlying services, provided 836 using data generated by IoT devices and access networks. Such 837 services include end device location, radio network information, 838 bandwidth management and congestion management (e.g., by the 839 congestion management feature of oneM2M [oneM2M-TR0052]). 841 Related challenges include: 843 * (Computation placement) Selecting, in a centralized or 844 distributed/peer-to-peer manner, an appropriate compute device 845 based on available resources, location of data input and data 846 sinks, compute node properties, etc., and with varying goals 847 including for example end-to-end latency, privacy, high 848 availability, energy conservation, or network efficiency, e.g. 849 using load balancing techniques to avoid congestion 851 * Onboarding code on a platform or computing device, and invoking 852 remote code execution, possibly as part of a distributed 853 programming model and with respect to similar concerns of latency, 854 privacy, etc. These operations should deal with heterogeneous 855 compute nodes [Schafer], and may in some cases also support end 856 devices, including IoT devices, as compute nodes [Larrea] 858 * Adapting Quality of Results (QoR) for applications where a perfect 859 result is not necessary [Li] 861 * Assisted or automatic partitioning of code 862 [I-D.sarathchandra-coin-appcentres] 864 * Supporting computation across trust domains, e.g. verifying 865 computation results 867 * Support for computation mobility: relocating an instance from one 868 compute node to another, while maintaining a given service level. 869 Session continuity when communicating with end devices that are 870 mobile, possibly at high speed (e.g. in vehicular scenarios). 871 Defining lightweight execution environments for secure code 872 mobility, e.g., using WebAssembly [Nieke] 874 * Defining, managing, and verifying Service Level Agreements (SLA) 875 for edge computing systems. Pricing is a related challenge 877 4.4.2. Edge Storage and Caching 879 Local storage or caching enable local data processing (e.g., pre- 880 processing or analysis), as well as delayed data transfer to the 881 cloud or delayed physical shipping. An edge node may offer local 882 data storage (where persistence is subject to retention policies), 883 caching, or both. Caching generally refers to temporary storage to 884 improve performance with no persistence guarantees. An edge caching 885 component manages data persistence, e.g., it schedules removal of 886 data when it is no longer needed. Other related aspects include 887 authenticating and encrypting data. Edge storage and caching can 888 take the form of a distributed storage system. 890 Related challenges include 892 * (Cache and data placement) Using cache positioning and data 893 placement strategies to minimize data retrieval delay [Liu], 894 energy consumption. Caches may be positioned in the access 895 network infrastructure, or on end devices 897 * Maintaining consistency, freshness, and privacy of stored/cached 898 data in systems that are distributed, constrained, and dynamic 899 (e.g. due to end devices and computing nodes churn or mobility). 900 For example, [Mortazavi] exploits a hierarchical storage 901 organization. Freshness-related metrics include the age of 902 information [Yates], that captures the timeliness of information 903 from a sender (e.g. an IoT device). 905 4.4.3. Northbound/Southbound Communication 907 An edge cloud may provide a northbound data plane or management plane 908 interface to a remote network, e.g., a cloud, home or enterprise 909 network. This interface does not exist in standalone (local-only) 910 scenarios. To support such an interface when it exists, an edge 911 computing component needs to expose an API, deal with authentication 912 and authorization, support secure communication. 914 An edge cloud may provide an API or interface to local or mobile 915 users, for example, to provide access to services and applications, 916 or to manage data published by local/mobile devices. 918 Edge computing nodes communicate with IoT devices over a southbound 919 interface, typically for data acquisition and IoT end device 920 management. 922 Related challenges include: 924 * Defining edge computing abstractions, such as PaaS [Yangui], 925 suitable for users and cloud systems to interact with edge 926 computing systems, and dealing with interoperability issues such 927 as data models heterogeneity. 929 4.4.4. Communication Brokering 931 A typical function of IoT edge computing is to facilitate 932 communication with IoT end devices: for example, enable clients to 933 register as recipients for data from devices, as well as forwarding/ 934 routing of traffic to or from IoT end devices, enabling various data 935 discovery and redistribution patterns, e.g., north-south with clouds, 936 east-west with other edge devices 937 [I-D.mcbride-edge-data-discovery-overview]. Another related aspect 938 is dispatching alerts and notifications to interested consumers both 939 inside and outside of the edge computing domain. Protocol 940 translation, analytics, and video transcoding may also be performed 941 when necessary. 943 Communication brokering may be centralized in some systems, e.g., 944 using a hub-and-spoke message broker, or distributed like with 945 message buses, possibly in a layered bus approach. Distributed 946 systems may leverage direct communication between end devices, over 947 device-to-device links. A broker can ensure communication 948 reliability, traceability, and in some cases transaction management. 950 Related challenges include: 952 * Enabling secure and resilient communication between IoT end 953 devices and remote cloud, e.g. through multipath support 955 4.5. Application Components 957 IoT edge computing can host applications such as the ones mentioned 958 in Section 2.4. While describing components of individual 959 applications is out of our scope, some of those applications share 960 similar functions, such as IoT end device management, data 961 management, described below. 963 4.5.1. IoT End Devices Management 965 IoT end device management includes managing information about the IoT 966 devices, including their sensors, how to communicate with them, etc. 967 Edge computing addresses the scalability challenges from the massive 968 number of IoT end devices by separating the scalability domain into 969 edge/local networks and remote networks. For example, in the context 970 of the oneM2M standard, the software campaign feature enables 971 installing, deleting, activating, and deactivating software 972 functions/services on a potentially large number of edge nodes 973 [oneM2M-TR0052]. Using a dash board or a management software, a 974 service provider issues those requests through an IoT cloud platform 975 supporting the software campaign functionality. 977 Challenges listed in Section 4.3.1 may be applicable to IoT end 978 devices management as well. 980 4.5.2. Data Management and Analytics 982 Data storage and processing at the edge is a major aspect of IoT edge 983 computing, directly addressing high-level IoT challenges listed in 984 Section 3. Data analysis such as performed in AI/ML tasks performed 985 at the edge may benefit from specialized hardware support on 986 computing nodes. 988 Related challenges include: 990 * Addressing concerns on resource usage, security, and privacy when 991 sharing, processing, discovering, or managing data. For example 992 by presenting data in views composed of an aggregation of related 993 data [Zhang]; protecting data communication between authenticated 994 peers [Basudan]; classifying data (e.g., in terms of privacy, 995 importance, validity, etc.); compressing and encrypting data, 996 e.g., using homomorphic encryption to directly process encrypted 997 data [Stanciu]. 999 * Other concerns on edge data discovery (e.g., streaming data, 1000 metadata, events) include siloization and lack of standard in edge 1001 environments that can be dynamic (e.g. vehicular networks) and 1002 heterogeneous [I-D.mcbride-edge-data-discovery-overview] 1004 * Data-driven programming models [Renart], e.g. event-based, 1005 including handling of naming and data abstractions 1007 * Addressing concerns such as limited resources, privacy, dynamic 1008 and heterogeneous environment, to deploy machine learning at the 1009 edge. For example, making machine learning more lightweight and 1010 distributed (e.g., to enable distributed inference at the edge), 1011 supporting shorter training time and simplified models, and 1012 supporting models that can be compressed for efficient 1013 communication [Murshed] 1015 * While edge computing can support IoT services independently of 1016 cloud computing, it can also be connected to cloud computing. 1017 Thus, the relationship of IoT edge computing to cloud computing, 1018 with regard to data management, is another potential challenge 1019 [ISO_TR] 1021 4.6. Simulation and Emulation Environments 1023 IoT Edge Computing brings new challenges to simulation and emulation 1024 tools used by researchers and developers. A varied set of 1025 applications, network, and computing technologies can coexist in a 1026 distributed system, which makes modeling difficult. Scale, mobility, 1027 and resource management are additional challenges [SimulatingFog]. 1029 Tools include simulators, where simplified application logic runs on 1030 top of a fog network model, and emulators, where actual applications 1031 can be deployed, typically in software containers, over a cloud 1032 infrastructure (e.g. Docker, Kubernetes) itself running over a 1033 network emulating network edge conditions such as variable delays, 1034 throughput and mobility events. To gain in scale, emulated and 1035 simulated systems can be used together in hybrid federation-based 1036 approaches [PseudoDynamicTesting], while to gain in realism physical 1037 devices can be interconnected with emulated systems. Examples of 1038 related work and platforms include the publicly accessible MEC 1039 sandbox work recently initiated in ETSI [ETSI_Sandbox], and open 1040 source simulators and emulators ([AdvantEDGE] emulator and tools 1041 cited in [SimulatingFog]). EdgeNet [Senel] is a globally distributed 1042 edge cloud for Internet researchers, using nodes contributed by 1043 institutions, and based on Docker for containerization and Kubernetes 1044 for deployment and node management. 1046 5. Security Considerations 1048 Privacy and security are drivers for the adoption of edge computing 1049 for IoT (Section 3.4). As discussed in Section 4.3.1, authentication 1050 and trust (between computing nodes, management nodes, end devices) 1051 can be challenging as scale, mobility, and heterogeneity increase. 1052 The sometimes disconnected nature of edge resources can prevent 1053 relying on a third-party authority. Distributed edge computing is 1054 exposed to issues with reliability and denial of service attacks. 1055 Personal or proprietary IoT data leakage is also a major threat, 1056 especially due to the distributed nature of the systems 1057 (Section 4.5.2). 1059 However, edge computing also brings solutions in the security space: 1060 maintaining privacy by computing sensitive data closer to data 1061 generators is a major use case for IoT edge computing. An edge cloud 1062 can be used to take actions based on sensitive data, or anonymizing, 1063 aggregating or compressing data prior to transmitting to a remote 1064 cloud server. Edge computing communication brokering functions can 1065 also be used to secure communication between edge and cloud networks. 1067 6. Acknowledgements 1069 The authors would like to thank Joo-Sang Youn, Akbar Rahman, Michel 1070 Roy, Robert Gazda, Rute Sofia, Thomas Fossati, Chonggang Wang, Marie- 1071 Jose Montpetit, Carlos J. Bernardos, Milan Milenkovic, Dale Seed, 1072 JaeSeung Song and Roberto Morabito for their valuable comments and 1073 suggestions on this document. 1075 7. Informative References 1077 [AdvantEDGE] 1078 "Mobile Edge Emulation Platform", Source Code Repository , 1079 2020, . 1081 [Anglano] Anglano, C., Canonico, M., Castagno, P., Guazzone, M., and 1082 M. Sereno, "A game-theoretic approach to coalition 1083 formation in fog provider federations", IEEE Third 1084 International Conference on Fog and Mobile Edge Computing 1085 (FMEC), pages 123-130 , 2018. 1087 [Ashton] Ashton, K., "That Internet of Things thing", RFID J. vol. 1088 22, no. 7, pp. 97-114 , 2009. 1090 [Basudan] Basudan, S., Lin, X., and K. Sankaranarayanan, "A privacy- 1091 preserving vehicular crowdsensing-based road surface 1092 condition monitoring system using fog computing", IEEE 1093 Internet of Things Journal, 4(3):772-782 , 2017. 1095 [Botta] Botta, A., Donato, W., Persico, V., and A. Pescape, 1096 "Integration of Cloud Computing and Internet of Things: A 1097 survey", Future Gener. Comput. Syst., vol. 56, pp. 1098 684-700 , 2016. 1100 [Cao] Cao, L., Merican, A., Zad Tootaghaj, D., Ahmed, F., 1101 Sharma, P., and V. Saxena, "ECaaS: A Management Framework 1102 of Edge Container as a Service for Business Workload", 4th 1103 International Workshop on Edge Systems, Analytics and 1104 Networking , 2021, 1105 . 1107 [Chen] Baotong Chen, ., Jiafu Wan, ., Antonio Celesti, ., Di Li, 1108 ., Haider Abbas, ., and . Qin Zhang, "Edge computing in 1109 IoT-based manufacturing", IEEE Communications Magazine , 1110 2018, . 1113 [Chiang] Chiang, M. and T. Zhang, "Fog and IoT: An overview of 1114 research opportunities", IEEE Internet Things J., vol. 3, 1115 no. 6, pp. 854-864 , 2016. 1117 [Cloudlets] 1118 Mahadev Satyanarayanan, ., Paramvir Bahl, ., Ramón 1119 Cáceres, ., and . Nigel Davies, "The Case for VM-Based 1120 Cloudlets in Mobile Computing", IEEE Pervasive Computing , 1121 2009, . 1123 [Echeverria] 1124 Echeverria, S., Klinedinst, D., Williams, K., and G. A 1125 Lewis, "Establishing trusted identities in disconnected 1126 edge environments", IEEE/ACM Symposium Edge Computing 1127 (SEC), pages 51-63. , 2016. 1129 [ENERGY] Beckel, C., Sadamori, L., Staake, T., and S. Santini, 1130 "Revealing Household Characteristics from Smart Meter 1131 Data", Energy, vol. 78, pp. 397-410 , 2014. 1133 [ETSI_MEC_01] 1134 ETSI, ., "Multi-access Edge Computing (MEC); Terminology", 1135 ETSI GS 001 , 2019, . 1138 [ETSI_MEC_03] 1139 ETSI, ., "Mobile Edge Computing (MEC); Framework and 1140 Reference Architecture", ETSI GS 003 , 2019, 1141 . 1144 [ETSI_Sandbox] 1145 "Multi-access Edge Computing (MEC) MEC Sandbox Work Item", 1146 Portal , 2020, 1147 . 1150 [Fan] Fan, Q. and N. Ansari, "Cost aware cloudlet placement for 1151 big data processing at the edge", IEEE International 1152 Conference on Communications (ICC), pages 1-6 , 2017. 1154 [Harchol] Harchol, Y., Mushtaq, A., McCauley, J., Panda, A., and S. 1155 Shenker, "Cessna: Resilient edge-computing", Workshop on 1156 Mobile Edge Communications, pages 1-6. ACM , 2018. 1158 [I-D-defoy-t2trg-iot-edge-computing-background] 1159 de Foy, X., Hong, J., Hong, Y., Kovatsch, M., Schooler, 1160 E., and D. Kutscher, "Machine learning at the network 1161 edge: A survey", draft-defoy-t2trg-iot-edge-computing- 1162 background-00 , 2020, . 1166 [I-D.bernardos-sfc-fog-ran] 1167 Bernardos, C. J. and A. Mourad, "Service Function Chaining 1168 Use Cases in Fog RAN", Work in Progress, Internet-Draft, 1169 draft-bernardos-sfc-fog-ran-10, 22 October 2021, 1170 . 1173 [I-D.ietf-core-oscore-groupcomm] 1174 Tiloca, M., Selander, G., Palombini, F., Mattsson, J. P., 1175 and J. Park, "Group OSCORE - Secure Group Communication 1176 for CoAP", Work in Progress, Internet-Draft, draft-ietf- 1177 core-oscore-groupcomm-13, 25 October 2021, 1178 . 1181 [I-D.mcbride-edge-data-discovery-overview] 1182 McBride, M., Kutscher, D., Schooler, E., Bernardos, C. J., 1183 Lopez, D. R., and X. D. Foy, "Edge Data Discovery for 1184 COIN", Work in Progress, Internet-Draft, draft-mcbride- 1185 edge-data-discovery-overview-05, 1 November 2020, 1186 . 1189 [I-D.sarathchandra-coin-appcentres] 1190 Trossen, D., Sarathchandra, C., and M. Boniface, "In- 1191 Network Computing for App-Centric Micro-Services", Work in 1192 Progress, Internet-Draft, draft-sarathchandra-coin- 1193 appcentres-04, 26 January 2021, 1194 . 1197 [ISO_TR] "Information Technology - Cloud Computing - Edge Computing 1198 Landscape", ISO/IEC TR 23188 , 2018. 1200 [Jeffery] Jeffery, A., Howard, H., and R. Mortier, "Rearchitecting 1201 Kubernetes for the Edge", 4th International Workshop on 1202 Edge Systems, Analytics and Networking , 2021, 1203 . 1205 [Jeong] Jeong, T., Chung, J., Hong, J.W., and S. Ha, "Towards a 1206 distributed computing framework for fog", IEEE Fog World 1207 Congress (FWC), pages 1-6 , 2017. 1209 [Jones] David Jones, ., Chris Snider, ., Aydin Nassehi, ., Jason 1210 Yon, ., and . Ben Hicks, "Characterising the Digital Twin: 1211 A systematic literature review", CIRP Journal of 1212 Manufacturing Science and Technology , 2020, 1213 . 1216 [Kelly] Kelly, R., "Internet of Things Data to Top 1.6 Zettabytes 1217 by 2022", 2015, 1218 . 1222 [Khan] Khan, L.U., Yaqoob, I., Tran, N.H., Kazmi, S.M.A., Dang, 1223 T.N., and C.S. Hong, "Edge Computing Enabled Smart Cities: 1224 A Comprehensive Survey", arXiv:1909.08747 , 2019. 1226 [Larrea] Larrea, J. and A. Barbalace, "The serverkernel operating 1227 system", Third ACM International Workshop on Edge Systems, 1228 Analytics and Networking , 2020, 1229 . 1231 [LFEDGE-EVE] 1232 Linux Foundation, ., "Project Edge Virtualization Engine 1233 (EVE)", Portal , 2020, 1234 . 1236 [Li] Li, Y., Chen, Y., Lan, T., and G. Venkataramani, "Mobiqor: 1237 Pushing the envelope of mobile edge computing via quality- 1238 of-result optimization", IEEE 37th International 1239 Conference on Distributed Computing Systems (ICDCS), pages 1240 1261-1270 , 2017. 1242 [Lin] Lin, J., Yu, W., Zhang, N., Yang, X., Zhang, H., and W. 1243 Zhao, "A survey on Internet of Things: Architecture, 1244 enabling technologies, security and privacy, and 1245 applications", IEEE Internet of Things J., vol. 4, no. 5, 1246 pp. 1125-1142 , 2017. 1248 [Liu] Liu, J., Bai, B., Zhang, J., and K.B. Letaief, "Cache 1249 placement in fog-rans: From centralized to distributed 1250 algorithms", IEEE Transactions on Wireless Communications, 1251 16(11):7039-7051 , 2017. 1253 [Mahadev] Satyanarayanan, M., "The Emergence of Edge Computing", 1254 Computer, vol. 50, no. 1, pp. 30-39 , 2017. 1256 [Mortazavi] 1257 Hossein Mortazavi, S., Balasubramanian, B., de Lara, E., 1258 and S.P. Narayanan, "Toward Session Consistency for the 1259 Edge", USENIX, Workshop on Hot Topics in Edge Computing 1260 (HotEdge 18) , 2018, 1261 . 1264 [Murshed] Murshed, M., Murphy, C., Hou, D., Khan, N., 1265 Ananthanarayanan, G., and F. Hussain, "Machine learning at 1266 the network edge: A survey", arXiv:1908.00080 , 2019. 1268 [Nieke] Nieke, M., Almstedt, L., and R. Kapitza, "Edgedancer: 1269 Secure Mobile WebAssembly Services on the Edge", 4th 1270 International Workshop on Edge Systems, Analytics and 1271 Networking , 2021, 1272 . 1274 [NIST] Mell, P. and T. Grance, "The NIST definition of Cloud 1275 Computing", Natl. Inst. Stand. Technol, vol. 53, no. 6, p. 1276 50 , 2009. 1278 [NVIDIA] Grzywaczewski, A., "Training AI for Self-Driving Vehicles: 1279 the Challenge of Scale", NVIDIA Developer Blog , 2017, 1280 . 1283 [oneM2M-TR0001] 1284 Mladin, C., "TR 0001, Use Cases Collection", oneM2M , 1285 October 2018, 1286 . 1289 [oneM2M-TR0018] 1290 Lu, C. and M. Jiang, "TR 0018, Industrial Domain 1291 Enablement", oneM2M , February 2019, 1292 . 1295 [oneM2M-TR0026] 1296 Yamamoto, K., Mladin, C., and V. Kueh, "TR 0026, Vehicular 1297 Domain Enablement", oneM2M , January 2020, 1298 . 1301 [oneM2M-TR0052] 1302 Yamamoto, K. and C. Mladin, "TR 0052, Study on Edge and 1303 Fog Computing in oneM2M systems", oneM2M , September 2020, 1304 . 1307 [oneM2M-TS0002] 1308 He, S., "TS 0002, Requirements", oneM2M , February 2019, 1309 . 1312 [OpenFog] "OpenFog Reference Architecture for Fog Computing", 1313 OpenFog Consortium , 2017. 1315 [PseudoDynamicTesting] 1316 Ficco, M., Esposito, C., Xiang, Y., and F. Palmieri, 1317 "Pseudo-Dynamic Testing of Realistic Edge-Fog Cloud 1318 Ecosystems", IEEE Communications Magazine, Nov. 2017 , 1319 2017. 1321 [Renart] Renart, E.G., Diaz-Montes, J., and M. Parashar, "Data- 1322 driven stream processing at the edge", IEEE 1st 1323 International Conference on Fog and Edge Computing 1324 (ICFEC), pages 31-40 , 2017. 1326 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 1327 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 1328 Acronym in the IETF", BCP 161, RFC 6291, 1329 DOI 10.17487/RFC6291, June 2011, 1330 . 1332 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 1333 Application Protocol (CoAP)", RFC 7252, 1334 DOI 10.17487/RFC7252, June 2014, 1335 . 1337 [RFC7390] Rahman, A., Ed. and E. Dijk, Ed., "Group Communication for 1338 the Constrained Application Protocol (CoAP)", RFC 7390, 1339 DOI 10.17487/RFC7390, October 2014, 1340 . 1342 [Schafer] Schafer, D., Edinger, J., VanSyckel, S., Paluska, J.M., 1343 and C. Becker, "Tasklets: Overcoming Heterogeneity in 1344 Distributed Computing Systems", IEEE 36th International 1345 Conference on Distributed Computing Systems Workshops 1346 (ICDCSW), Nara, pp. 156-161 , 2016. 1348 [Senel] Senel, B., Mouchet, M., Cappos, J., Fourmaux, O., 1349 Friedman, T., and R. McGeer, "EdgeNet: A Multi-Tenant and 1350 Multi-Provider Edge Cloud", 4th International Workshop on 1351 Edge Systems, Analytics and Networking , 2021, 1352 . 1354 [Shi] Shi, W., Cao, J., Zhang, Q., Li, Y., and L. Xu, "Edge 1355 computing: vision and challenges", IEEE Internet of Things 1356 J., vol. 3, no. 5, pp. 637-646 , 2016. 1358 [SimulatingFog] 1359 Svorobej, S. and . al, "Simulating Fog and Edge Computing 1360 Scenarios: An Overview and Research Challenges", MPDI 1361 Future Internet 2019 , 2019. 1363 [Stanciu] Stanciu, V., van Steen, M., Dobre, C., and A. Peter, 1364 "Privacy-Preserving Crowd-Monitoring Using Bloom Filters 1365 and Homomorphic Encryption", 4th International Workshop on 1366 Edge Systems, Analytics and Networking , 2021, 1367 . 1369 [Weiner] Weiner, M., Jorgovanovic, M., Sahai, A., and B. Nikolie, 1370 "Design of a low-latency, high-reliability wireless 1371 communication system for control applications", IEEE Int. 1372 Conf. Commun. (ICC), Sydney, NSW, Australia, pp. 1373 3829-3835 , 2014. 1375 [Yangui] Yangui, S., Ravindran, P., Bibani, O., H Glitho, R., Ben 1376 Hadj-Alouane, N., Morrow, M.J., and P.A. Polakos, "A 1377 platform as-a-service for hybrid cloud/fog environments", 1378 IEEE International Symposium on Local and Metropolitan 1379 Area Networks (LANMAN), pages 1-7 , 2016. 1381 [Yates] Yates, R.D. and S.K. Kaul, "The Age of Information: Real- 1382 Time Status Updating by Multiple Sources", IEEE 1383 Transactions on Information Theory, vol. 65, no. 3, pp. 1384 1807-1827 , 2019. 1386 [Yousefpour] 1387 Yousefpour, A., Fung, C., Nguyen, T., Kadiyala, K., 1388 Jalali, F., Niakanlahiji, A., Kong, J., and J.P. Jue, "All 1389 one needs to know about fog computing and related edge 1390 computing paradigms: A complete survey", Journal of 1391 Systems Architecture, vol. 98, pp. 289-330 , 2019. 1393 [Zhang] Zhang, Q., Zhang, X., Zhang, Q., Shi, W., and H. Zhong, 1394 "Firework: Big data sharing and processing in 1395 collaborative edge environment", Fourth IEEE Workshop on 1396 Hot Topics in Web Systems and Technologies (HotWeb), pages 1397 20-25 , 2016. 1399 [Zhang2] Zhang, J., Chen, B., Zhao, Y., Cheng, X., and F. Hu, "Data 1400 Security and Privacy-Preserving in Edge Computing 1401 Paradigm: Survey and Open Issues", IEEE Access, vol. 6, 1402 pp. 18209-18237 , 2018. 1404 [_60802] IEC/IEEE, ., "Use Cases IEC/IEEE 60802 V1.3", IEC/IEEE 1405 60802 , 2018, . 1408 Authors' Addresses 1410 Jungha Hong 1411 ETRI 1412 218 Gajeong-ro, Yuseung-Gu 1413 Daejeon 1414 34129 1415 Republic of Korea 1417 Email: jhong@etri.re.kr 1419 Yong-Geun Hong 1420 Daejeon University 1421 62 Daehak-ro, Dong-gu 1422 Daejeon 1423 300716 1424 Republic of Korea 1426 Email: yonggeun.hong@gmail.com 1428 Xavier de Foy 1429 InterDigital Communications, LLC 1430 1000 Sherbrooke West 1431 Montreal H3A 3G4 1432 Canada 1434 Email: xavier.defoy@interdigital.com 1435 Matthias Kovatsch 1436 Huawei Technologies Duesseldorf GmbH 1437 Riesstr. 25 C // 3.OG 1438 80992 Munich 1439 Germany 1441 Email: ietf@kovatsch.net 1443 Eve Schooler 1444 Intel 1445 2200 Mission College Blvd. 1446 Santa Clara, CA, 95054-1537 1447 United States of America 1449 Email: eve.m.schooler@intel.com 1451 Dirk Kutscher 1452 University of Applied Sciences Emden/Leer 1453 Constantiaplatz 4 1454 26723 Emden 1455 Germany 1457 Email: ietf@dkutscher.net