Video Explanation​

Industrial Embodied Intelligence Dataset(CNCF'LFX 25@kubeedge8739 )Project Explanation

YOLO Detection Predictions​

Real-world inference results on test images showing deformable component detection:

Model Performance Analysis​

Optimal F1-score achieved at confidence threshold of ~0.5	High precision maintained across various confidence levels

Dataset Analysis​

Distribution of component classes across the training dataset	Spatial correlation analysis between different component types

Why Embodied Intelligence Benchmarking for Industrial Manufacturing?​

The manufacturing industry is experiencing a profound transformation driven by intelligent robotics, adaptive production lines, and precision assembly systems. Modern industrial environments demand more than basic task execution,they require multimodal perception, force-controlled manipulation, and real-time quality verification integrated into cohesive workflows.

A First-of-Its-Kind Benchmark Addressing Critical Gaps​

Despite growing interest in Embodied Intelligence for manufacturing, the research community faces a critical shortage of benchmarks that evaluate real-world industrial assembly scenarios. Existing work falls short in several key areas:

Current Limitations in the Field:​

• Isolated Task Focus: Popular benchmarks like YCB and OCRTOC evaluate individual capabilities (grasping, object detection) but don't assess how these integrate in multi-stage industrial workflows
• Missing Deformable Component Coverage: While NIST has developed benchmarks for rigid components and simple deformable objects (cables, belts), existing datasets don't support multiple representations of deformable objects with different self-occlusions typical in manufacturing PubMed Central
• No Electronic Assembly Benchmarks: Despite datasets for PCB defect detection (BoardVision, FICS-PCB) and component recognition (ElectroCom61), none evaluate robotic assembly of electronic components, particularly those requiring precision force control
• Academic-Industry Gap: A large gap exists between embodied AI in academic research and what manufacturers can feasibly implement National Institute of Standards and Technology, with limited benchmarks that mirror real production challenges

What Makes This Project Unique:​

This work represents the first comprehensive benchmark for robotic assembly of deformable electronic components,a scenario ubiquitous in modern electronics manufacturing yet completely absent from existing research infrastructure. Our contributions are unprecedented in scope:

1. First Multimodal Industrial Assembly Dataset: To our knowledge, no publicly available dataset combines RGB-D vision, force/torque sensor data, and robot trajectories for electronic component assembly.

2. First Deformable Electronics Benchmark: Existing deformable object benchmarks focus on textiles (GarmentLab), food items, or abstract shapes. This is the first benchmark specifically designed for deformable electronic components like flexible circuits and memory modules,components that require both precision alignment and compliance control.

3. First End-to-End Assembly Workflow Evaluation: Unlike fragmented benchmarks that test perception OR manipulation OR verification separately, we evaluate complete assembly pipelines. This mirrors how industrial systems actually operate, where failure in any stage cascades to overall task failure.

4. Practical Manufacturing Relevance: Our five assembly scenarios (RAM modules, cooling mounts, CPU sockets, flexible circuits, security chips) represent actual production challenges in electronics manufacturing,an industry segment worth over $2 trillion globally that currently relies heavily on manual assembly due to lack of validated automation solutions.

Addressing Real Industrial Needs​

The electronics manufacturing sector faces mounting pressure to automate assembly operations involving:

• Sub-millimeter positioning tolerances (CPU socket assembly: 55% accuracy in our baseline)
• Deformable components requiring force adaptation (flexible circuits: 70% accuracy baseline)
• Quality verification at component scale (current detection: 91.19% mAP50)

Manufacturers need streamlined ways of assessing the productive impact of AI systems through AI-specific productivity metrics and test methods National Institute of Standards and Technology. This benchmark directly addresses that need by providing:

• Realistic Industrial Scenarios: Deformable component assembly representing real manufacturing challenges
• Comprehensive Multimodal Dataset: RGB-D images, force/torque sensor data, and robot trajectories,the complete sensor suite used in production environments
• End-to-End Evaluation: Metrics that assess entire workflows, not just individual subtasks, revealing integration challenges invisible to component-level testing
• Reproducible Infrastructure: Standardized test environments and baseline algorithms that enable fair comparison across different robotic systems

By combining distributed AI benchmarking framework KubeEdge-Ianvs with this domain-specific industrial dataset, we enable objective comparison of robotic systems on tasks that mirror real-world manufacturing complexity,bridging the gap between academic research and industrial deployment while accelerating the development of reliable autonomous assembly systems.

Contents:​

• Prerequisites
• Dataset Setup
• Benchmark Configuration
• Algorithm Integration
• Analyze Results
• Ianvs Installation
• Run Benchmarking

Prerequisites​

System Requirements:​

• One machine (laptop or VM) with 4+ CPUs
• 8GB+ RAM (depends on simulation complexity)
• 20GB+ free disk space
• Internet connection for downloads
• Python 3.8+ installed (Python 3.9 recommended)
• CUDA-capable GPU recommended for YOLOv8 (optional but accelerates training)

Knowledge Requirements:​

• Basic understanding of Python and machine learning
• Familiarity with robotic manipulation concepts
• Understanding of computer vision fundamentals

Note: This benchmark has been tested on Linux platforms. Windows users may need to adapt commands accordingly.

Dataset Setup​

The Deformable Component Assembly Dataset is a comprehensive multimodal dataset specifically designed for industrial assembly benchmarking. It contains:

• RGB and depth images from simulated PyBullet cameras
• Force/torque sensor data from robot wrist measurements
• YOLO-format annotations for object detection training
• Assembly success/failure labels for end-to-end evaluation
• Robot trajectory logs capturing full motion sequences

Download the Dataset​

Access the dataset from Kaggle:

# Download from: https://www.kaggle.com/datasets/kubeedgeianvs/deformable-assembly-dataset

mkdir -p /ianvs/datasets
cd /ianvs/datasets

Transfer the downloaded .zip file to the datasets folder and extract:

unzip deformable_assembly_dataset.zip

Dataset Structure​

The dataset contains 2,227 frames across 5 episodes, each focusing on different component types:

deformable_assembly_dataset/
├── episodes/
│   ├── episode_001_ram/              # 451 frames - RAM module assembly
│   │   ├── images/
│   │   │   ├── rgb/                  # RGB camera frames
│   │   │   ├── depth/                # Depth maps
│   │   │   └── segmentation/         # Segmentation masks
│   │   ├── labels/                   # YOLO format annotations
│   │   ├── sensor_data/              # Force/torque logs
│   │   │   ├── force_torque_log.csv
│   │   │   └── robotic_arm_poses.csv
│   │   └── metadata/
│   ├── episode_002_cooling_mounts/   # 400 frames
│   ├── episode_003_cpu_slot/         # 400 frames
│   ├── episode_004_fcp/              # 400 frames - Flexible circuits
│   └── episode_005_chip_key/         # 577 frames
├── index/                            # Train/test splits
│   ├── train_index1.txt
│   ├── test_index1.txt
│   └── ...
└── dataset_info.json

Each episode represents a distinct assembly challenge:

Total Dataset Size: ~830 MB

Algorithm Integration​

The benchmark implements a multi-stage single-task paradigm that orchestrates three sequential modules:

1. Perception Module (perception.py)​

Uses YOLOv8 for real-time component detection:

• Identifies component locations on the assembly panel
• Provides bounding boxes with confidence scores
• Extracts orientation information for grasping

2. Manipulation Module (manipulation.py)​

Implements force-controlled assembly operations:

• Executes precise grasping based on detection results
• Uses haptic feedback for delicate component handling
• Adapts to deformation during flexible circuit placement
• Monitors force/torque to prevent damage

3. Verification Module (perception.py)​

CNN-based visual quality inspection:

• Captures final assembly state
• Detects misalignments and defects
• Generates pass/fail classifications

The entire workflow is orchestrated by naive_assembly_process.py, which executes these stages sequentially for each component type.

Results​

Results are available in the output directory (/ianvs/industrialEI_workspace) as defined in benchmarkingjob.yaml.

Comprehensive Performance Report​

╔═══════════════════════════════════════════════════════════════════════════╗
║                    🎯 EVALUATION COMPLETE                                 ║
║                                                                           ║
║  ► Overall Accuracy Score: 6400                                          ║
║  ► Overall Accuracy Percentage: 64.00%                                   ║
║  ► Detection mAP50: 91.19%                                               ║
║  ► Assembly Success Rate: 83.33%                                         ║
║  ► Total Frames Processed: 150                                           ║
║  ► Successful Assemblies: 125                                            ║
║                                                                           ║
║  🏆 Final Combined Score: 0.7759 (77.59%)                                ║
╚═══════════════════════════════════════════════════════════════════════════╝

Per-Component Performance Metrics​

┌───────────────┬────────┬──────┬───────┬───────┬───────┬───────┬────────┬───────┐
│ Component     │ Frames │ Imgs │ Prec% │ Rec%  │ mAP50 │ mAP95 │ Acc%   │ Succ% │
├───────────────┼────────┼──────┼───────┼───────┼───────┼───────┼────────┼───────┤
│ ram           │ 30     │ 136  │ 94.1  │ 73.5  │ 89.4  │ 62.5  │ 60.0   │ 83.3  │
│ cooling_mounts│ 30     │ 120  │ 99.6  │ 99.2  │ 99.5  │ 91.8  │ 65.0   │ 83.3  │
│ cpu_slot      │ 30     │ 150  │ 99.3  │ 80.0  │ 82.5  │ 72.4  │ 55.0   │ 83.3  │
│ fcp           │ 30     │ 150  │ 96.1  │ 81.7  │ 88.2  │ 75.9  │ 70.0   │ 83.3  │
│ chip_key      │ 30     │ 120  │ 99.6  │ 99.2  │ 99.5  │ 95.7  │ 70.0   │ 83.3  │
╞═══════════════╪════════╪══════╪═══════╪═══════╪═══════╪═══════╪════════╪═══════╡
║ OVERALL       ║ 150    ║ 676  ║ 97.7  ║ 85.9  ║ 91.2  ║ 78.8  ║ 64.0   ║ 83.3  ║
└───────────────┴────────┴──────┴───────┴───────┴───────┴───────┴────────┴───────┘

Metric Definitions:​

• Frames: Number of test frames processed
• Images: Number of training images used
• Prec%: Detection precision (true positives / predicted positives)
• Rec%: Detection recall (true positives / actual positives)
• mAP50: Mean Average Precision at IoU threshold 0.50
• mAP95: Mean Average Precision at IoU 0.50:0.95 range
• Acc%: Assembly accuracy (weighted combination of position, orientation, deformation control, and force feedback)
• Succ%: Binary assembly success rate

Training Performance Analysis​

The benchmark provides comprehensive training visualizations:

Matplotlib curves:

Model Performance & Dataset Analysis​

Metric Type	Visualization 1	Visualization 2
Confidence Analysis	Optimal F1-score achieved at confidence threshold of ~0.5	High precision maintained across various confidence levels
Trade-off Analysis	Trade-off analysis between precision and recall metrics	Recall performance across different detection thresholds
Dataset Insights	Distribution of component classes across the training dataset	Spatial correlation analysis between different component types

YOLO Detection Predictions​

Real-world inference results on test images showing deformable component detection:

Real-world inference results on test images showing deformable component detection:

Model successfully detects RAM modules, cooling mounts, CPU slots, FCPs, and chip keys with accurate bounding boxes*

Key training insights:

• F1-Score vs Confidence: Optimal performance at ~0.5 confidence threshold
• Precision-Recall Trade-off: High precision maintained across recall ranges
• Loss Convergence: Stable training with consistent loss reduction

Validation Results​

These visualizations demonstrate the model's capability to accurately detect and localize components across diverse assembly scenarios.

Key Achievements and Community Impact​

This embodied intelligence benchmarking framework delivers significant value to the industrial AI community:

1. Comprehensive Multimodal Dataset​

A publicly available dataset containing:

• 2,227 annotated frames across 5 component types
• RGB-D images with precise calibration
• Force/torque sensor logs with millisecond precision
• Complete robot trajectory data
• Assembly success labels validated against industrial standards

2. End-to-End Evaluation Infrastructure​

Unlike existing benchmarks that assess isolated capabilities, this framework evaluates:

• Complete multi-stage workflows from perception to verification
• Integration quality between vision and force control
• Real-world assembly success rates
• System robustness across component variations

3. Reproducible Baseline Algorithms​

Production-ready implementations including:

• YOLOv8-based component detection achieving 91.19% mAP50
• Force-controlled manipulation with haptic feedback
• CNN visual inspection for quality assurance
• Complete workflow orchestration framework

4. Standardized Performance Metrics​

Industry-relevant metrics that capture:

• Detection accuracy (Precision, Recall, mAP)
• Assembly quality (position, orientation, deformation control)
• Overall success rates (83.33% baseline)
• Combined performance scores for leaderboard rankings

This benchmark enables fair comparison of different embodied intelligence approaches, accelerating innovation in industrial automation.

Technical Insights and Future Directions​

The benchmark reveals important insights about industrial embodied intelligence:

Current Capabilities:​

• High detection performance (97.7% precision, 85.9% recall overall)
• Consistent success rates across component types (83.3%)
• Robust handling of deformable components (FCP assembly: 70% accuracy)

Improvement Opportunities:​

• RAM module assembly shows lower recall (73.5%), indicating detection challenges with reflective surfaces
• CPU slot assembly has reduced accuracy (55%), suggesting the need for finer position control
• Deformable component handling requires enhanced force feedback algorithms

Future Enhancements​

The framework is designed to accommodate advanced features:

• Intelligent decision-making: Conditional logic for accessory selection based on assembly configuration
• Failure recovery: Adaptive re-planning when initial assembly attempts fail
• Complex workflows: Multi-robot coordination for larger assemblies
• Transfer learning: Cross-component generalization to reduce training requirements

For more technical details and ongoing development, see the project repository and subscribe to updates.

How to Enable Embodied Intelligence Benchmarking with KubeEdge-Ianvs​

The following procedures set up a complete benchmarking environment for industrial assembly tasks using KubeEdge-Ianvs with custom multimodal datasets.

After completing these steps, you'll have a fully operational benchmark capable of evaluating multi-stage robotic assembly workflows. Ianvs readme

Getting Started with Your Own Benchmarks​

Ianvs Installation​

1. Download the Ianvs codebase​

git clone https://github.com/kubeedge/ianvs.git
cd ianvs

2. Create and activate a virtual environment​

sudo apt-get install -y virtualenv
mkdir ~/venv 
virtualenv -p python3 ~/venv/ianvs
source ~/venv/ianvs/bin/activate

3. Install system dependencies and Python packages​

sudo apt-get update
sudo apt-get install libgl1-mesa-glx -y
python -m pip install --upgrade pip

python -m pip install ./examples/resources/third_party/*
python -m pip install -r requirements.txt

4. Install benchmark-specific requirements​

pip install -r examples/industrialEI/single_task_learning_bench/deformable_component_manipulation/requirements.txt

5. Install Ianvs​

python setup.py install  
ianvs -v

If the version information prints successfully, Ianvs is ready to use.

Benchmark Configuration​

The benchmark is organized following the Ianvs framework structure:

ianvs/examples/industrialEI/single_task_learning_bench/
└── deformable_assembly/
    ├── testalgorithms/
    │   └── assembly_alg/
    │       ├── components/
    │       │   ├── perception.py          # YOLOv8 detection + CNN inspection
    │       │   └── manipulation.py        # Force control algorithms
    │       ├── testenv/
    │       │   ├── acc.py                 # End-to-end metrics
    │       │   └── testenv.yaml           # Environment configuration
    │       └── naive_assembly_process.py  # Workflow orchestration
    ├── benchmarkingjob.yaml
    └── README.md

Configure Dataset Paths​

The dataset path is pre-configured in testenv.yaml. Set up the algorithm path:

export PYTHONPATH=$PYTHONPATH:/ianvs/examples/industrialEI/single_task_learning_bench/deformable_assembly/testalgorithms/assembly_alg

The configuration defines the complete test environment including:

• Dataset locations and splits
• Simulation parameters
• Sensor configurations
• Performance metrics

Run Benchmarking​

Execute the benchmark from the Ianvs root directory:

cd /ianvs
ianvs -f examples/industrialEI/single_task_learning_bench/deformable_assembly/benchmarkingjob.yaml

The benchmark will:

1. Load the multimodal dataset with all sensor modalities
2. Run YOLOv8 detection on each frame for component localization
3. Execute force-controlled assembly in PyBullet simulation
4. Perform CNN-based visual inspection on assembled products
5. Calculate comprehensive performance metrics
6. Generate detailed reports and leaderboards

This framework is designed for extensibility. To create your own industrial assembly benchmarks:

1. Collect your dataset: Use the provided PyBullet simulation scripts or capture real robot data
2. Define your metrics: Customize testenv/acc.py for scenario-specific evaluation
3. Implement algorithms: Follow the module interface in components/ for perception and manipulation
4. Configure Ianvs: Adapt testenv.yaml and benchmarkingjob.yaml for your environment
5. Run and compare: Execute benchmarks and contribute results to the community leaderboard

The complete documentation, dataset access, and baseline implementations are available in the Ianvs repository.

Conclusion​

This work is part of the KubeEdge-Ianvs distributed synergy AI benchmarking initiative, advancing the state of cloud-edge collaborative intelligence for industrial applications.

Author: Ronak Raj

1.22 What's New​

• Add Hold/Release Mechanism for Controlling Edge Resource Updates
• Beehive Framework Upgrade, Supporting Configurable Submodule Restart Policies
• Device Model Update Based On Thing Model and Product Concept
• Add Featuregates for Pod Resources Server and CSI Plugin in EdgeCore Integrated Lightweight Kubelet
• C language Mapper-Framework Support
• Upgrade Kubernetes Dependency to v1.31.12

Release Highlights​

Add Hold/Release Mechanism for Controlling Edge Resource Updates​

In applications such as autonomous driving, drones, and robotics, we want to control when updates to edge resources occur, ensuring that these resources cannot be updated without the permission of the edge administrator. In v1.22.0, we introduced a hold/release mechanism to control updates to edge resources.

On the cloud side, users can indicate that the corresponding Pod should be held on the edge by adding the annotation edge.kubeedge.io/hold-upgrade: "true" to resources like Deployment, StatefulSet, and DaemonSet.

On the edge, Pods marked with edge.kubeedge.io/hold-upgrade: "true" will be temporarily held and not processed. Edge administrators can release the lock on the Pod to complete the resource update by executing the following command:

keadm ctl unhold-upgrade pod <pod-name>

Alternatively, they can execute the following command to unlock all held edge resources on the edge node:

keadm ctl unhold-upgrade node

Refer to the link for more details.(#6348, #6418)

Beehive Framework Upgrade, Supporting Configurable Submodule Restart Policies​

In release 1.17, we implemented auto-restart for the EdgeCore modules, allowing global configuration of edge modules restarts. In release 1.22, we optimized the Beehive framework to support restart policy configurations for edge submodules. We also standardized the error handling for starting Beehive submodules.

Refer to the link for more details.(#6444, #6445)

Device Model Update Based On Thing Model and Product Concept​

The current Device Model is designed based on the thing model concept. In traditional IoT, devices are usually designed with a three-tier structure: thing model, product, and device instance, which can lead to user confusion during actual use.

In release 1.22, we upgraded the device model design by integrating the concepts of thing models and actual products. We extracted the protocolConfigData and visitors fields from existing device instances into the device model, allowing device instances to share these model configurations. Additionally, to reduce the cost of separating models, device instances can override these configurations.

Refer to the link for more details.(#6457, #6458)

Add Featuregates for Pod Resources Server and CSI Plugin in EdgeCore Integrated Lightweight Kubelet​

In previous versions, we removed the Pod Resources Server capability from the integrated lightweight Kubelet in EdgeCore. However, in some use cases, users wish to restore this capability for monitoring Pods. Additionally, the default activation of the CSI Plugin in Kubelet can lead to failures in offline environments due to failed CSINode creation when starting EdgeCore.

In v1.22.0, we added featuregates for the Pod Resources Server and CSI Plugin in the lightweight Kubelet. If you need to enable the Pod Resources Server or disable the CSI Plugin, you can add the following feature gates to your EdgeCore configuration:

apiVersion: edgecore.config.kubeedge.io/v1alpha2
kind: EdgeCore
modules:
     edged:
         tailoredKubeletConfig:
            featureGates:
               KubeletPodResources: true
               DisableCSIVolumePlugin: true
...

Refer to the link for more details.(kubeedge/kubernetes#12, kubeedge/kubernetes#13, #6452)

C language Mapper-Framework Support​

In v1.20.0, we added a Java version of the Mapper-Framework based on the existing Go language version. Due to the diversity of communication protocols for edge IoT devices, many edge device driver protocols are implemented in C. Thus, in the new release, KubeEdge offers a C language version of the Mapper-Framework. Users can access the feature-multilingual-mapper-c branch in the KubeEdge main repository to generate custom Mapper projects in C using the Mapper Framework.

Refer to the link for more details.(#6405, #6455)

Upgrade Kubernetes Dependency to v1.31.12​

Upgrade the vendored kubernetes version to v1.31.12, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details.(#6443)

1.21 What's New​

• New Generation Node Task API and Implementation
• Support Closed Loop Flow Control
• Support Update Edge Configuration from Cloud
• Support One-Click Deployment of Dashboard and Integration of kubeedge/keink

Release Highlights​

New Generation Node Task API and Implementation​

In v1.21, we redesigned the status structure and operation process of node jobs to track error information and facilitate developers' understanding. In the new design, the node job status includes Phase (Init, InProgress, Completed, Failure) and nodeStatus. The nodeStatus consists of Phase (Pending, InProgress, Successful, Failure, Unknown), actionFlow, nodeName, reason, and business-related fields. A YAML example of the NodeUpgradeJob status is provided below.

status:
  nodeStatus:
    - actionFlow:
        - action: Check
          status: 'True'
          time: '2025-05-28T08:12:01Z'
        - action: WaitingConfirmation
          status: 'True'
          time: '2025-05-28T08:12:01Z'
        - action: Backup
          status: 'True'
          time: '2025-05-28T08:12:01Z'
        - action: Upgrade
          status: 'True'
          time: '2025-05-28T08:13:02Z'
      currentVersion: v1.21.0
      historicVersion: v1.20.0
      nodeName: ubuntu
      phase: Successful
  phase: Completed

Refer to the link for more details.(#6082, #6084)

Support Closed Loop Flow Control​

In v1.21, we have optimized the traffic closed-loop function of node groups. Applications within a node group can only access application services within the same group and unable to access services of other node groups. With this mechanism, users can easily achieve network isolation between multiple edge regions, ensuring that application services in different regions do not interfere with each other.

Refer to the link for more details.(#6097, #6077)

Support Update Edge Configuration from Cloud​

In many cases, cloud-based direct updates to EdgeCore configuration files for edge nodes offer greater convenience than manual updates from edge node, especially for batch operations that boost efficiency by managing multiple nodes simultaneously.

In v1.21.0, ConfigUpdateJob CRD is introduced to allows users to update configuration files for edge nodes in the cloud. The updateFields within the CRD is used to specify the configuration items that need to be updated.

CRD Sample:

apiVersion: operations.kubeedge.io/v1alpha2
kind: ConfigUpdateJob
metadata:
  name: configupdate-test
spec:
  failureTolerate: "0.3"
  concurrency: 1
  timeoutSeconds: 180
  updateFields:
    modules.edgeStream.enable: "true"
  labelSelector:
    matchLabels:
      "node-role.kubernetes.io/edge": ""
      node-role.kubernetes.io/agent: ""

Refer to the link for more details.(#6338)

Support One-Click Deployment of Dashboard and Integration of kubeedge/keink​

In v1.21, the dashboard functionality has been enhanced by designing a BFF (Backend for Frontend) layer to connect the frontend user interface layer with the KubeEdge backend API. Additionally, the dashboard is integrated with the keink project, allowing users to launch a keink cluster in the dashboard environment with just one command to experience KubeEdge features.

Refer to the link for more details.(#50)

Important Steps before Upgrading​

• From v1.21, the v1alpha2 node job enables by default, and the CRD definition will be backward compatible. If you want to continue to use the v1alpha1 version of the NodeUpgradeJob and ImagePrePullJob, please setting the feature gates of ControllerManager and CloudCore.
  • Add a command arg for ControllerManager:

    --feature-gates=disableNodeTaskV1alpha2
    
  • Modify the CloudCore configuration:

    apiVersion: cloudcore.config.kubeedge.io/v1alpha2
    kind: CloudCore
    featureGates:
      disableNodeTaskV1alpha2: true
    ...
    

The v1alpha1 node jobs will no longer be maintained, and relevant codes will be clean up after v1.23. In addition, the node job has become a default disabled Beehive module in EdgeCore. If you want to use the node jobs, please modify the edgecore.yaml to enable it.

  modules:
    ...
    taskManager:
      enable: true

• From v1.21, keadm upgrade related commands(backup, upgrade, rollback) at the edge have been adjusted.
  • The upgrade command will not automatically execute the backup. The backup command needs to be triggered manually.
  • The upgrade command hides business-related flags and relevant codes will be cleaned up after v1.23.
  • All upgrade related commands use level 3 commands:

  keadm edge upgrade
  keadm edge backup
  keadm edge rollback
  

1.20 What's New​

• Support Batch Node Process
• Multi-language Mapper-Framework Support
• Support Pods logs/exec/describe and Devices get/edit/describe Operation at Edge Using keadm ctl
• Decouple EdgeApplications from NodeGroups, Support Node Label Selector
• CloudHub-EdgeHub Supports IPv6
• Upgrade Kubernetes Dependency to v1.30.7

Release Highlights​

Support Batch Node Process​

Previously, the keadm tool of KubeEdge only supports manual single-node management. However, in edge scenarios, the scale of nodes is often very large, and the management process of a single node can no longer cope with such large-scale scenarios.

In v1.20, we have provided the batch node operation and maintenance capability. With this capability, users only need to use one configuration file to perform batch operation and maintenance on all edge nodes through a control node (which can log in to all edge nodes). The supported operation and maintenance capabilities include join, reset, and upgrade.

# Configuration Requirements
keadm:
  download:
    enable: true              # <Optional> Whether to download the keadm package, which can be left unconfigured, default is true. if it is false, the 'offlinePackageDir' will be used.
    url: ""                   # <Optional> The download address of the keadm package, which can be left unconfigured. If this parameter is not configured, the official github repository will be used by default.
  keadmVersion: ""            # <Required> The version of keadm to be installed. for example: v1.19.0
  archGroup:                  # <Required> This parameter can configure one or more of amd64/arm64/arm.
    - amd64
  offlinePackageDir: ""       # <Optional> The path of the offline package. When download.enable is true, this parameter can be left unconfigured.
  cmdTplArgs:                 # <Optional> This parameter is the execution command template, which can be optionally configured and used in conjunction with nodes[x].keadmCmd.
    cmd: ""                   # This is an example parameter, which can be used in conjunction with nodes[x].keadmCmd.
    token: ""                 # This is an example parameter, which can be used in conjunction with nodes[x].keadmCmd.
nodes:
  - nodeName: edge-node       # <Required> Unique name, used to identify the node
    arch: amd64               # <Required> The architecture of the node, which can be configured as amd64/arm64/arm
    keadmCmd: ""              # <Required> The command to be executed on the node, can used in conjunction with keadm.cmdTplArgs. for example: "{{.cmd}} --edgenode-name=containerd-node1 --token={{.token}}"
    copyFrom: ""              # <Optional> The path of the file to be copied from the local machine to the node, which can be left unconfigured.
    ssh:
      ip: ""                  # <Required> The IP address of the node.
      username: root          # <Required> The username of the node, need administrator permissions.
      port: 22                # <Optional> The port number of the node, the default is 22.
      auth:                   # Log in to the node with a private key or password, only one of them can be configured.
        type: password        # <Required> The value can be configured as 'password' or 'privateKey'.
        passwordAuth:         # It can be configured as 'passwordAuth' or 'privateKeyAuth'.
          password: ""        # <Required> The key can be configured as 'password' or 'privateKeyPath'.
maxRunNum: 5                  # <Optional> The maximum number of concurrent executions, which can be left unconfigured. The default is 5.`

# Configuration Example
keadm:
  download:
    enable: true
    url: https://github.com/kubeedge/kubeedge/releases/download/v1.20.0 # If this parameter is not configured, the official github repository will be used by default
  keadmVersion: v1.20.0
  archGroup: # This parameter can configure one or more of amd64\arm64\arm
    - amd64
  offlinePackageDir: /tmp/kubeedge/keadm/package/amd64  # When download.enable is true, this parameter can be left unconfigured
  cmdTplArgs: # This parameter is the execution command template, which can be optionally configured and used in conjunction with nodes[x].keadmCmd
    cmd: join --cgroupdriver=cgroupfs --cloudcore-ipport=192.168.1.102:10000 --hub-protocol=websocket --certport=10002 --image-repository=docker.m.daocloud.io/kubeedge --kubeedge-version=v1.20.0 --remote-runtime-endpoint=unix:///run/containerd/containerd.sock
    token: xxx
nodes:
  - nodeName: ubuntu1   # Unique name
    arch: amd64
    keadmCmd: '{{.cmd}} --edgenode-name=containerd-node1 --token={{.token}}' # Used in conjunction with keadm.cmdTplArgs
    copyFrom: /root/test-keadm-batchjoin  # The file directory that needs to be remotely accessed to the joining node
    ssh:
      ip: 192.168.1.103
      username: root
      auth:
        type: privateKey   # Log in to the node using a private key
        privateKeyAuth:
          privateKeyPath: /root/ssh/id_rsa
  - nodeName: ubuntu2
    arch: amd64
    keadmCmd: join --edgenode-name=containerd-node2 --cgroupdriver=cgroupfs --cloudcore-ipport=192.168.1.102:10000 --hub-protocol=websocket --certport=10002 --image-repository=docker.m.daocloud.io/kubeedge --kubeedge-version=v1.20.0 --remote-runtime-endpoint=unix:///run/containerd/containerd.sock  # Used alone
    copyFrom: /root/test-keadm-batchjoin
    ssh:
      ip: 192.168.1.104
      username: root
      auth:
        type: password
        passwordAuth:
          password: *****
maxRunNum: 5

# Usage
keadm batch -c config.yaml

Refer to the link for more details.(#5988, #5968)

Multi-language Mapper-Framework Support​

To further reduce the complexity of developing custom Mapper, in this version, KubeEdge provides the Java version of Mapper-Framework. Users can access the KubeEdge feature-multilingual-mapper branch to use Mapper-Framework to generate a Java version of custom Mapper project.

Refer to the link for more details.(#5773, #5900)

Support Pods logs/exec/describe and Devices get/edit/describe Operation at Edge Using keadm ctl​

In v1.17, a new command keadm ctl has been introduced to support pods query and restart at Edge. In this release, keadm ctl supports more functionality including pod logs/exec/describe and device get/edit/describe to help users operate resources at edge, especially in offline scenarios.

[root@edgenode1 ~]# keadm ctl -h
Commands operating on the data plane at edge

Usage:
  keadm ctl [command]

Available Commands:
...
  describe    Show details of a specific resource
  edit        Edit a specific resource
  exec        Execute command in edge pod
  get         Get resources in edge node
  logs        Get pod logs in edge node
...

Refer to the link for more details.(#5752, #5901)

Decouple EdgeApplications from NodeGroups, Support Node Label Selector​

EdgeApplication can be overrides deployment spec(i.e. replicas, image, commands and environments) via the node group, and pod traffics are closed-loop in a node group(Deployments managed by EdgeApplication share a Service). But in the real scenario, the scope of nodes that need batch operations is different from that of nodes that need to collaborate with each other.

We add a new targetNodeLabels field for node label selectors in the EdgeApplication CRD, this field will allow the application to deploy based on node labels and apply overrides specific to those nodes.

apiVersion: apps.kubeedge.io/v1alpha1
kind: EdgeApplication
metadata:
  name: edge-app
  namespace: default
spec:
  replicas: 3
  image: my-app-image:latest
  # New field: targetNodeLabels
  targetNodeLabels:
    - labelSelector:
        - matchExpressions:
            - key: "region"
              operator: In
              values:
                - "HangZhou"
    overriders:
      containerImage:
        name: new-image:latest
      resources:
        limits:
          cpu: "500m"
          memory: "128Mi"

Refer to the link for more details.(#5755, #5845)

CloudHub-EdgeHub Supports IPv6​

We provide a configuration guide in the documentation on the official website, which is how KubeEdge enables the cloud-edge hub to support IPv6 in a K8s cluster.

Refer to the document https://kubeedge.io/docs/advanced/support_ipv6

Upgrade Kubernetes Dependency to v1.30.7​

Upgrade the vendered kubernetes version to v1.30.7, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#5997)

Important Steps before Upgrading​

• From v1.20, the default value for the EdgeCore configuration option edged.rootDirectory will change from /var/lib/edged to /var/lib/kubelet. If you wish to continue using the original path, you can set --set edged.rootDirectory=/var/lib/edged when installing EdgeCore with keadm.

In this article, we will introduce how to deploy and use the KubeEdge Dashboard.

Environment Pre-requisites​

We can obtain the source code of KubeEdge Dashboard from the KubeEdge Dashboard GitHub repository. Before building and deploying KubeEdge Dashboard, please ensure the following environment is set up:

• KubeEdge Cluster: Please refer to the KubeEdge official documentation to set up a KubeEdge cluster. KubeEdge Dashboard requires KubeEdge v1.15 or later versions.
• Node.js: Install Node.js on your system, it is recommended to use Node.js v18 or later versions.
• Node.js Package Manager: Install a Node.js package manager, such as npm, yarn, or pnpm.

Building and Deploying​

Once the environment is set up and the KubeEdge Dashboard source code has been downloaded, we can use the Node.js package manager to start the KubeEdge Dashboard. In the following instructions, we will use pnpm as the example to show how to install dependencies and run KubeEdge Dashboard.

First of all, we need to install the dependencies:

pnpm install

KubeEdge Dashboard interacts with KubeEdge resources via the Kubernetes API. Therefore, we need to set the API_SERVER environment variable to specify the API Server address:

pnpm run build
API_SERVER=https://192.168.33.129:6443 pnpm run start

After starting KubeEdge Dashboard, open http://localhost:3000 in your browser to access the dashboard.

For the KubeEdge cluster with self-signed certificates, we need to set the NODE_TLS_REJECT_UNAUTHORIZED=0 environment variable to bypass certificate verification:

NODE_TLS_REJECT_UNAUTHORIZED=0 API_SERVER=<api-server> pnpm run start

Creating a Login Token​

To authenticate with KubeEdge Dashboard, we need to create a token for login. The following instructions show how to create a service account dashboard-user in the kube-system namespace and generate a token for authentication.

First, we need to create a service account in the Kubernetes cluster:

kubectl create serviceaccount dashboard-user -n kube-system

To grant permissions to the service account, we need to create a cluster role binding that associates the service account with a cluster role. Kubernetes provides some built-in cluster roles, such as cluster-admin, which has access to all resources in the cluster. We can also refer to the Kubernetes documentation to create a custom cluster role if needed.

kubectl create clusterrolebinding dashboard-user-binding --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-user -n kube-system

Since Kubernetes v1.24, secrets for service accounts are no longer created automatically. We need to create an associated token by the kubectl create token command. The lifetime of the token will be determined by the server automatically, and we can specify the lifetime of the token by the --duration option.

kubectl create token dashboard-user -n kube-system

For Kubernetes v1.23 and earlier versions, Kubernetes automatically creates a secret for the service account. We can retrieve the secret by the kubectl describe secret command:

kubectl describe secret -n kube-system $(kubectl get secret -n kube-system | grep dashboard-user | awk '{print $1}')

Conclusion​

With KubeEdge Dashboard, we can more easily manage KubeEdge resources such as edge applications and devices. We will continue to enhance and optimize the KubeEdge Dashboard and user experience in future releases. We also welcome feedback and suggestions from the community.

For more information on KubeEdge Dashboard, please refer to the KubeEdge Dashboard GitHub repository.

1.19 What's New​

• Support Edge Nodes Report Event
• Support OTA(Over-The-Air) Upgrades for Edge Nodes
• Mapper Supports Device Data Writing
• Add OpenTelemetry to Mapper-framework
• A New Release of KubeEdge Dashboard

Release Highlights​

Support Edge Nodes Report Event​

Kubernetes Event serve as a report of an event somewhere in the cluster, reflecting status changes of cluster resources such as Nodes and Pods. In v1.19, EdgeCore supports reporting events to cloud, allowing users to directly access the status of edge nodes or Pods in the cloud via kubectl get events or kubectl describe {resource_type} {resource_name}.

This feature is disabled by default in v1.19. To enable it, execute --set modules.edged.reportEvent=true when install EdgeCore with keadm or modify the EdgeCore configuration file and then restart EdgeCore.

Refer to the link for more details.(#5722, #5811)

Support OTA(Over-The-Air) Upgrades for Edge Nodes​

On the basis of NodeUpgradeJob upgrade, we add the edge node confirmation card point and the validation of the image digest. The card point confirmation allows the node upgrade to be delivered to the edge side, and the upgrade can be performed only after the user is confirmed. Image digest validation can ensure that the kubeedge/installation-pacakge image to be upgraded is secure and reliable at the edge side.

In v1.19, we can use spec.imageDigestGatter in NodeUpgradeJob to define how to get the image digest. The value to directly define the digest, The registryAPI to get the mirror digest via registry v2 API, both are mutually exclusive. If none is configured, the image digest is not verified during the upgrade.

We can also use spec.requireConfirmation to configure requireConfirmation for NodeUpgradeJob to determine whether we want to confirm at the edge side.

Refer to the link for more details.(#5589, #5761, #5863)

Mapper Supports Device Data Writing​

In v1.19, we add the ability to write device data in Mapper-Framework. User can use device methods through the API provided by Mapper and complete data writing to device properties.

• Device method API

A new definition of device methods is added in new release. Users can define device methods in the device-instance file that can be called by the outside world in device. Through device methods, users can control and write data to device properties.

• Device data writing

In v1.19, the Mapper API capability is improved and a new device method interface is added. The user can use the relevant interface to obtain all the device methods contained in a device, as well as the calling command of the device method. Through the returned calling command, user can create a device write request to write data to device.

Refer to the link for more details.(#5662, #5902)

Add OpenTelemetry to Mapper-framework​

In v1.19, we add the OpenTelemetry observability framework to mapper data plane, which can encapsulate device data and push data to multiple types of applications or databases. This feature can enhance the mapper data plane's ability to push device data.

Refer to the link for more details.(#5628)

A New Release of KubeEdge Dashboard​

Based on previous Dashboard release, we have refactored the KubeEdge Dashboard using the more popular frameworks Next.js and MUI. In the new release, we rewrote and optimized around 60 pages and components, reducing about 70% of redundant code. We also upgraded the KubeEdge and native Kubernetes APIs to the latest version to maintain compatibility and added TypeScript definitions for the APIs.

Refer to the link for more details.(#29)

Important Steps before Upgrading​

• In the next release (v1.20), the default value for the EdgeCore configuration option edged.rootDirectory will change from /var/lib/edged to /var/lib/kubelet. If you wish to continue using the original path, you can set --set edged.rootDirectory=/var/lib/edged when installing EdgeCore with keadm.

• In v1.19, please use --kubeedge-version to specify the version when installing KubeEdge with keadm, --profile version is no longer supported.

We would like to thank our TOC sponsors Cathy Zhang and Lin Sun, and everyone who has collaborated over the past six years on KubeEdge’s design, development, and deployment.

KubeEdge is an open source edge computing project built on Kubernetes, which expands the cloud native ecosystem to scenarios and industries outside of data centers. It extends Kubernetes’ native container orchestration and scheduling capabilities to the edge and provides capabilities such as edge application management, cloud-edge metadata synchronization, and edge IoT device management.

KubeEdge was open sourced by Huawei Cloud in November 2018. It was accepted as the first cloud native edge project in the CNCF Sandbox project in 2019 and was promoted to an incubating project in September 2020.

Since joining CNCF, the project team has grown to include maintainers from 15 organizations and has attracted over 1,600 contributors from more than 35 countries and 110 organizations. Contributing organizations include Huawei Cloud, DaoCloud, HarmonyCloud, China Unicom, Inovex, ARM, QingCloud Technology, BoCloud, China Mobile, Inspur, Sony, China Telecom, Zhejiang Lab, SEL Laboratory of Zhejiang University, Beijing University of Posts and Telecommunications, and University of Electronic Science and Technology of China.

“KubeEdge started with a vision to extend cloud native technology to the edge and foster a robust edge-cloud computing ecosystem. Today, it has been adopted across numerous innovative and unforeseen fields. This achievement is a testament to the incredible creativity and dedication of our contributors, whose efforts have driven the project’s success,” said Kevin Wang and Yin Ding, KubeEdge TSC members. “Graduation marks a new beginning, and we are excited to explore even greater possibilities for the entire cloud native ecosystem.”

KubeEdge has been widely applied in various industries, including CDN, intelligent transportation, smart energy, smart retail, smart parks, automobiles, intelligent logistics, finance, power, and blockchain. It has been deployed in industry-leading projects such as the largest cloud native cloud-edge collaborative highway toll station management project, the first cloud native satellite-ground collaborative satellite, the first cloud native electric vehicle, and more.

“Kubernetes was born through usage across massive data centers but has evolved, just like Linux, for new environments and has become a leading platform in edge computing,” said Chris Aniszczyk, CTO, CNCF. “KubeEdge has been at the forefront of this transformation, bringing Kubernetes into new frontiers from electric cars to outer space. We look forward to seeing where the KubeEdge project goes next as a graduate project.”

Supporting Quotes:​

Huawei

“Since it was open sourced, KubeEdge has received the attention and support of industry partners and end users. It has achieved breakthrough innovative practices in industry fields such as intelligent transportation, finance, energy, electric vehicles, robotics, and logistics. The graduation of KubeEdge will further promote the cloud native digital transformation of enterprises and release greater industrial value. As a pioneer and popularizer of cloud native technology, Huawei Cloud will continue to cooperate with CNCF and the community in the future to jointly promote the development of the cloud native industry.”

– Bruno Zhang, CTO, Huawei Cloud

DaoCloud

“The graduation of KubeEdge marks a significant milestone in the field of Cloud Native Edge computing. As the official first edge computing project of CNCF, KubeEdge provides robust computing and management support for resource-constrained edge devices, and its graduation signifies the technological maturity and industry recognition of Cloud Native Edge computing. Since the launch of the KubeEdge project, the DaoCloud team has actively participated, and contributed our extensive expertise on project governance/core tech development and customer promotion in manufacturing and retail sectors. In the future, we will continue to collaborate with the KubeEdge community to provide more flexible, secure, and efficient solutions to global enterprises and developers, and shape a new era of smart edges.”

– Hongbing Zhang, COO, DaoCloud

HarmonyCloud

“Congratulations on the successful graduation of KubeEdge open source project! As a member of KubeEdge Technical Steering Committee (TSC) team, it has been a pleasure to witness the journey of KubeEdge from incubation to graduation. From discovery, trial use and issue raising, pull request raising, feature raising, application and promotion, KubeEdge is an important component in the core technology framework of the cloud edge computing team to provide customers with edge computing product delivery. We have edge computing landing cases in telecommunication, finance, transportation, energy, bridges, logistics, military and other fields. KubeEdge has always kept its design clean and architecture clear, thanks to every open source contributor, bringing great flexibility to applications across the edge computing industry. In the future, HarmonyCloud will also continuously participate in the KubeEdge community and jointly promote the development of the cloud native edge computing industry to release the ubiquitous value of cloud native!”

– Huan Wei, Senior Director & Head of Edge Computing, HarmonyCloud

QingCloud

“Since it was open sourced, KubeEdge has led the trend of cloud native edge computing with its unique architecture, exerting huge influence and demonstration effect. While promoting the application of cloud native technology, it also promotes the development of edge computing. With the adoption of increasingly heavyweight industry users and the participation of numerous community contributors worldwide, it is an honor to witness the entire process of KubeEdge from incubation to graduation as a member of the KubeEdge Technical Steering Committee (TSC). Congratulations to the KubeEdge project on becoming the first edge computing project to graduate from CNCF! As the core maintainer of KubeSphere, while deeply participating in the KubeEdge project, QingCloud Technology has also been actively promoting the implementation of KubeEdge in various industries. We aim to promote the development of edge computing together with many domestic and foreign counterparts and community users!”

– Benjamin Huo, Lead of Observability and Edge Computing, QingCloud

ByteDance

“It is a great pleasure to witness the graduation of KubeEdge from CNCF. This milestone not only embodies the collective efforts of the community but also signals that cloud native edge computing has entered a new stage of maturity. In the future, we will continue to closely cooperate with global developers and partners to promote the innovation and application of edge computing technology and accelerate the digital transformation of various industries.”

– Tina Tsou, KubeEdge Technical Steering Committee (TSC) Member, ByteDance

BoCloud

“KubeEdge is the longest-lasting and most active project of edge computing architecture software. It has an extremely profound influence on the industry and has been widely adopted in the intelligent transformation of industries such as industry, transportation, and equipment manufacturing, making positive contributions to the promotion of the intelligent manufacturing plan. The KubeEdge community has attracted many open source contributors and cooperation projects with an active and open attitude. The open source edge network and storage components such as FabEdge and Carina from BoCloud have achieved good compatibility and project applications, and multiple project practices of cloud-edge-end AI model transfer, model conversion and loading have been implemented. The graduation of KubeEdge indicates the arrival of the mature stage of edge computing technology and the period of rapid industrial development. It is believed that the project will definitely gain more practical applications in the future. BoCloud is willing to jointly promote industrialization development with the project community and support the implementation of more scenarios.”

– Ji Cui, Vice President, BoCloud

Sony

“It is the most exciting news to see the graduation of KubeEdge from CNCF, which shouts out production-ready Kubernetes edge solutions. KubeEdge extends the orchestration capability down to edge environments to provide unified cluster management and sophisticated edge-specific features.

We are confident to see KubeEdge be applied to many more use cases in the near future, especially for edge computing such as robotics distributed systems, and AI/ML application deployment at edge. Not only development but also community engagement and growth are really important for open source based activity. KubeEdge also has been doing great work to encourage the developers and users to be engaged in the community, that really works to design the KubeEdge that can support broad use cases and requirements at edge environment and business logics. We will continue to engage and support KubeEdge development and community.”

– Tomoya FUjita, Software Engineer, Sony Corporation of America

NIO

“As a platform specifically developed for cloud-edge collaboration, KubeEdge can effectively address the challenges such as computing resource limited, a vast number of edge nodes, and differences in operating environments that the automotive field faces when applying the cloud native technology stack. After extensive research work, we have built NIO’s entire vehicle-cloud collaboration platform with KubeEdge as the core and used it in mass-produced vehicle models for the first time, bringing significant improvements in development and delivery efficiency, team collaboration, and so on. It will also achieve ultra-large-scale vehicle management. Hereby, we congratulate KubeEdge on its graduation. We also look forward to having deeper cooperation with the community, continuing to promote the development of the KubeEdge community, and continuously building innovative applications of cloud native solutions in the revolutionary EV field.”

– Saint Jiang, Architect of Digital Systems, Strategic New Business, NlO

Zeblok

“With KubeEdge’s graduation from CNCF, the edge computing landscape reaches a new frontier. At Zeblok Computational Inc., we’re thrilled to incorporate KubeEdge into our Ai-MicroCloud, delivering cutting-edge AI capabilities directly to the edge. This synergy drives innovation where real-time intelligence matters most.”– Vikarna Tathe, Platform Engineering Lead, Zeblok Computational Inc.

OEHI

“KubeEdge officially reaching the next level of maturity at CNCF is a strong recognition of the successful development efforts around this technology. The Open Edge and HPC Initiative (OEHI) strongly believes in the potential of KubeEdge as a key element of upcoming compute continuum infrastructures. To train people for this new paradigm of computing infrastructure design, OEHI co-organised a tutorial at ISC HPC 2024 where participants could get first-hand experience with KubeEdge. In the future, we will continue to promote this technology.”

– Dirk Pleiter, Chair, OEHI

SF Technology

“The graduation of the KubeEdge is exciting. It is a rising star in the field of cloud native edge computing. SF Technology has been deeply involved in the logistics field for many years. KubeEdge is like a powerful assistant for us to move towards intelligence. From the efficient operation of logistics sorting to the full life cycle processing of transportation links, the edge computing capabilities provided by KubeEdge help us create a smarter and more efficient logistics system. In its development process, we have witnessed its growth and transformation, and we have also actively integrated it into our business scenarios. SF Technology looks forward to walking hand in hand with KubeEdge and continuously exploring new possibilities in the future, pushing the intelligence of the logistics industry to a new height and allowing edge computing to play a greater value in the circulation of every package.”

– Panggang Cheng, Lead of Edge Cloud Container Product, SF Technology

In the future, the KubeEdge community will keep the open governance model as well as the collaborative concept, and will further improve the user experience, providing a more reliable and stable experience for users. Meanwhile, KubeEdge will continue to explore new areas, such as cloud edge collaborative AI, cloud edge collaborative robots, and edge cluster management.

• To learn more about KubeEdge:
• Check out the project website and GitHub repository.
• Read the documentation.
• Join the community Slack.
• Follow the project on X.

1.18 What's New​

• Router Manager Support High Availability
• Authorization Enhancement for CloudCore Websocket API
• Support Device Status Reporting
• Keadm Tool Enhancement
• Encapsulate Token, CA and Certificate operations
• Upgrade Kubernetes Dependency to v1.29.6

Release Highlights​

Router Manager Support High Availability(HA)​

When CloudCore adopts high availability deployment, RouterManager needs to determine whether to route messages to the correct CloudCore. This feature is already supported in v1.18.0, and RouterManager supports high availability.

Refer to the link for more details.(#5619, #5635)

Authorization Enhancement for CloudCore Websocket API​

CloudCore need restrict the access to cluster resources for edge nodes. In this release，CloudCore supports node authorization mode. CloudHub identify the sender of messages and check whether the sender has sufficient permissions, so that CloudCore can restrict an edge node from operating the resources owned by other edge nodes.

Refer to the link for more details.(#5512, #5585)

Support Device Status Reporting​

Device status reporting is a capability required for device management. It was previously planned but not implemented. In version 1.18, we support this feature. Device status reporting can be easily implemented based on the community mapper template.

Refer to the link for more details.(#5649, #5650)

Keadm Tool Enhancement​

Before this release, keadm(KubeEdge Installation Tool) is only supported to configure a subset of parameters before EdgeCore was installed. Now we can use the '--set' flag to configure the parameters of the full configuration edgecore.yaml file, so that users can customize the parameters at installation time, without having to modify the configuration and restart the service after installation.

Refer to the link for more details.(#5564, #5574)

Encapsulate Token, CA and Certificate operations​

We refactor the token and certificate-related codes, summarize the same businesses, and abstract the ability of certificates to improve scalability, maintainability, and readability.

Refer to the link for more details.(#5502, #5544)

Upgrade Kubernetes Dependency to v1.29.6​

Upgrade the vendered kubernetes version to v1.29.6, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#5656)

Important Steps before Upgrading​

• The CloudCore Authorization feature is disabled by default in release 1.18. If you need to use this feature, please set cloudhub.authorization.enable=true.

Why Cilium for KubeEdge​

Cilium is the one of the most advanced and efficient container network interface plugin for Kubernetes, that provides network connectivity and security for containerized applications in Kubernetes clusters. It leverages eBPF (extended Berkeley Packet Filter) technology to implement networking and security policies at the Linux kernel level, allowing for high-performance data plane operations and fine-grained security controls.

And KubeEdge extends the cluster orchestration capability down to edge environments to provide unified cluster management and sophisticated edge specific features.

Enabling Cilium with KubeEdge allows us to take advantage of both benefits even for edge computing environments. We can deploy the application containers where EdgeCore running and bind Cilium to connect with workloads in the cloud infrastructure. This is because Cilium can also enable WireGuard VPN with transparent encryption of traffic between Cilium-managed endpoints.

Further more, we can also rely on Cilium Tetragon Security Observability and Runtime Enforcement to confine security risk and vulnerability in edge environment.

How to enable Cilium with KubeEdge​

The following procedures to set up a simple cluster system with Kubernetes and KubeEdge with Cilium. Since this is a new approach and still beta phase, the following manual operations are required to do so.

After all the operations, we can develop the following cluster configuration with KubeEdge with Cilium.

• Prerequisites
• Kubernetes Master Setup
• Cilium Install and Setup
• KubeEdge CloudCore Setup
• KubeEdge EdgeCore Setup
• Check Cilium Connectivity from Pods

Prerequisites​

• KubeEdge Release v1.16 or later required.

  To enable Cilium with KubeEdge, we must use KubeEdge Release v1.16 or later. This is because that cilium-agent needs to issue InClusterConfig APIs to Kubernetes API server to configure cilium-agent. This should be no problem with Kubernetes nodes, but with KubeEdge those API requests and responses need to be bypassed via KubeEdge MetaManager. You can see KubeEdge EdgeCore supports Cilium CNI for more details.

• Compatible Kubernetes version with KubeEdge Release v1.16.

  You can find compatible and supported Kubernetes version here.

• It Requires super user rights (or root rights) to run commands.

Kubernetes Master Setup​

Refer to KubeEdge Setup Prerequisites, and set up the Kubernetes API server as following.

### Check node status
> kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE   VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
tomoyafujita   Ready    control-plane   25s   v1.26.15   AA.BBB.CCC.DD   <none>        Ubuntu 20.04.6 LTS   5.15.0-102-generic   containerd://1.6.32

### Taint this node so that CloudCore can be deployed on the control-plane
> kubectl taint node tomoyafujita node-role.kubernetes.io/control-plane:NoSchedule-
node/tomoyafujita untainted
> kubectl get nodes -o json | jq '.items[].spec.taints'
null

Cilium Install and Setup​

Refer to Cilium Quick Installation, install and set up cilium deployments in the cluster.

> cilium version
cilium-cli: v0.16.9 compiled with go1.22.3 on linux/amd64
cilium image (default): v1.15.5
cilium image (stable): v1.15.5
cilium image (running): unknown. Unable to obtain cilium version. Reason: release: not found

and install Cilium with enabling wireguard VPN in the cluster,

> cilium install --set encryption.enabled=true --set encryption.type=wireguard --set encryption.wireguard.persistentKeepalive=60
...

> cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 1
                       cilium-operator    Running: 1
Cluster Pods:          1/2 managed by Cilium
Helm chart version:
Image versions         cilium             quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 1

Add nodeAffinity for Cilium DaemonSet to make sure these pods are only created on cloud nodes. These Cilium pods are generic Cilium DaemonSet, so supposed to be running on cloud nodes but where EdgeCore running.

### Edit Cilium DaemonSet with the following patch
>  kubectl edit ds -n kube-system cilium

diff --git a/cilium-kubelet.yaml b/cilium-kubelet.yaml
index 21881e1..9946be9 100644
--- a/cilium-kubelet.yaml
+++ b/cilium-kubelet.yaml
@@ -29,6 +29,12 @@ spec:
         k8s-app: cilium
     spec:
       affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                - key: node-role.kubernetes.io/edge
+                  operator: DoesNotExist
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:

After editing, Cilium pods will be restarted.

KubeEdge CloudCore Setup​

1st of all, we need to install Keadm with the official procedure Installing KubeEdge with Keadm.

In this blog, we use Keadm v1.16.1 as following.

### Install v1.16.1 keadm command
> wget https://github.com/kubeedge/kubeedge/releases/download/v1.16.1/keadm-v1.16.1-linux-amd64.tar.gz
> tar -zxvf keadm-v1.16.1-linux-amd64.tar.gz
> cp keadm-v1.16.1-linux-amd64/keadm/keadm /usr/local/bin

> keadm version
version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"bd7b42acbfbe3a453c7bb75a6bb8f1e8b3db7415", GitTreeState:"clean", BuildDate:"2024-03-27T02:57:08Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

and then, start the CloudCore with v1.16.1.

> keadm init --advertise-address="AA.BBB.CCC.DD" --profile version=v1.16.1 --kube-config=/root/.kube/config
Kubernetes version verification passed, KubeEdge installation will start...
CLOUDCORE started
=========CHART DETAILS=======
NAME: cloudcore
LAST DEPLOYED: Tue Jun  4 17:19:15 2024
NAMESPACE: kubeedge
STATUS: deployed
REVISION: 1

After CloudCore is started, we also need to enable DynamicControllers.

### edit ConfigMap of CloudCore to enable dynamicController
> kubectl edit cm -n kubeedge cloudcore
> kubectl delete pod -n kubeedge --selector=kubeedge=cloudcore

### Check ConfigMap
> kubectl get cm -n kubeedge cloudcore -o yaml | grep "dynamicController" -A 1
      dynamicController:
        enable: true

To manage and handle the APIs from MetaManager that are originally come from Cilium running in the edge nodes, we need to give access permission to CloudCore by editing clusterRole and clusterRolebinding.

clusterRole:

### Edit and apply the following patch
> kubectl edit clusterrole cilium

diff --git a/cilium-clusterrole.yaml b/cilium-clusterrole.yaml
index 736e35c..fd5512e 100644
--- a/cilium-clusterrole.yaml
+++ b/cilium-clusterrole.yaml
@@ -66,6 +66,7 @@ rules:
   verbs:
   - list
   - watch
+  - get
 - apiGroups:
   - cilium.io
   resources:

clusterRolebinding:

### Edit and apply the following patch
> kubectl edit clusterrolebinding cilium

diff --git a/cilium-clusterrolebinding.yaml b/cilium-clusterrolebinding.yaml
index 9676737..ac956de 100644
--- a/cilium-clusterrolebinding.yaml
+++ b/cilium-clusterrolebinding.yaml
@@ -12,3 +12,9 @@ subjects:
 - kind: ServiceAccount
   name: cilium
   namespace: kube-system
+- kind: ServiceAccount
+  name: cloudcore
+  namespace: kubeedge
+- kind: ServiceAccount
+  name: cloudcore
+  namespace: default

Finally, we get the token after CloudCore has restarted.

> keadm gettoken
<TOKEN_HASH>

KubeEdge EdgeCore Setup​

With the token provided above, we can start the EdgeCore to join the cluster system.

> keadm join --cloudcore-ipport=AA.BBB.CCC.DD:10000 --kubeedge-version=v1.16.1 --cgroupdriver=systemd --token <TOKEN_HASH>
...<snip>
I0604 21:36:31.040859 2118064 join_others.go:265] KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -xe
I0604 21:36:41.050154 2118064 join.go:94] 9. Install Complete!

> systemctl status edgecore
● edgecore.service
     Loaded: loaded (/etc/systemd/system/edgecore.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-06-04 21:36:31 PDT; 40s ago
   Main PID: 2118341 (edgecore)
      Tasks: 24 (limit: 18670)
     Memory: 31.8M
        CPU: 849ms
     CGroup: /system.slice/edgecore.service
             └─2118341 /usr/local/bin/edgecore

After EdgeCore is started, we need to enable ServiceBus and MetaServer to edit edgecore.yaml.

### Edit and apply the following patch
> vi /etc/kubeedge/config/edgecore.yaml

### Restart edgecore systemd-service
> systemctl restart edgecore

diff --git a/edgecore.yaml b/edgecore.yaml
index 8d17418..5391776 100644
--- a/edgecore.yaml
+++ b/edgecore.yaml
@@ -62,6 +62,8 @@ modules:
       cgroupDriver: systemd
       cgroupsPerQOS: true
       clusterDomain: cluster.local
+      clusterDNS:
+      - 10.96.0.10
       configMapAndSecretChangeDetectionStrategy: Get
       containerLogMaxFiles: 5
       containerLogMaxSize: 10Mi
@@ -151,7 +151,7 @@ modules:
     enable: true
     metaServer:
       apiAudiences: null
-      enable: false
+      enable: true
       server: 127.0.0.1:10550
       serviceAccountIssuers:
       - https://kubernetes.default.svc.cluster.local
@@ -161,7 +161,7 @@ modules:
       tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
     remoteQueryTimeout: 60
   serviceBus:
-    enable: false
+    enable: true
     port: 9060
     server: 127.0.0.1
     timeout: 60

And then, we need to create another DaemonSet of cilium-agent only for EdgeCore nodes. We need to deploy cilium-agent pods to the nodes where KubeEdge EdgeCore runs and labeled with kubernetes.io/edge=. Besides that, cilium-agent needs to query the APIs to MetaServer instead of Kubernetes API-server that is required to keep the edge autonomy provided by KubeEdge.

### Dump original Cilium DaemonSet configuration
> kubectl get ds -n kube-system cilium -o yaml > cilium-edgecore.yaml

### Edit and apply the following patch
> vi cilium-edgecore.yaml

### Deploy cilium-agent aligns with edgecore
> kubectl apply -f cilium-edgecore.yaml

diff --git a/cilium-edgecore.yaml b/cilium-edgecore.yaml
index bff0f0b..3d941d1 100644
--- a/cilium-edgecore.yaml
+++ b/cilium-edgecore.yaml
@@ -8,7 +8,7 @@ metadata:
     app.kubernetes.io/name: cilium-agent
     app.kubernetes.io/part-of: cilium
     k8s-app: cilium
-  name: cilium
+  name: cilium-kubeedge
   namespace: kube-system
 spec:
   revisionHistoryLimit: 10
@@ -29,6 +29,12 @@ spec:
         k8s-app: cilium
     spec:
       affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                - key: node-role.kubernetes.io/edge
+                  operator: Exists
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
@@ -39,6 +45,8 @@ spec:
       containers:
       - args:
         - --config-dir=/tmp/cilium/config-map
+        - --k8s-api-server=127.0.0.1:10550
+        - --auto-create-cilium-node-resource=true
         - --debug
         command:
         - cilium-agent
@@ -178,7 +186,9 @@ spec:
       dnsPolicy: ClusterFirst
       hostNetwork: true
       initContainers:
-      - command:
+      - args:
+        - --k8s-api-server=127.0.0.1:10550
+        command:
         - cilium
         - build-config
         env:

We can see below that cilium-pq45v (normal cilium-agent pod) running on cloud node, and cilium-kubeedge-kkb7z (edgecore specific DaemonSet) is running with edgecore.

> kubectl get pods -A -o wide
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES
kube-system   cilium-kubeedge-kkb7z                  1/1     Running   0          32s     XX.YYY.ZZZ.WWW   edgemaster     <none>           <none>
kube-system   cilium-operator-fdf6bc9f4-445p6        1/1     Running   0          3h40m   AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   cilium-pq45v                           1/1     Running   0          3h32m   AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   coredns-787d4945fb-2bbdf               1/1     Running   0          8h      10.0.0.104       tomoyafujita   <none>           <none>
kube-system   coredns-787d4945fb-nmd2p               1/1     Running   0          8h      10.0.0.130       tomoyafujita   <none>           <none>
kube-system   etcd-tomoyafujita                      1/1     Running   0          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-apiserver-tomoyafujita            1/1     Running   1          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-controller-manager-tomoyafujita   1/1     Running   0          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-proxy-qmxqp                       1/1     Running   0          19m     XX.YYY.ZZZ.WWW   edgemaster     <none>           <none>
kube-system   kube-proxy-v2ht7                       1/1     Running   0          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-scheduler-tomoyafujita            1/1     Running   1          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kubeedge      cloudcore-df8544847-6mlw2              1/1     Running   0          4h23m   AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kubeedge      edge-eclipse-mosquitto-9cw6r           1/1     Running   0          19m     XX.YYY.ZZZ.WWW   edgemaster     <none>           <none>

Check Cilium Connectivity from Pods​

Now Cilium is ready to be used for application pods and containers to provide network connectivity. We can use busybox DaemonSet as following to try ping the network connectivity via Cilium.

> cat busybox.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - image: busybox
        command: ["sleep", "3600"]
        imagePullPolicy: IfNotPresent
        name: busybox

> kubectl apply -f busybox.yaml
daemonset.apps/busybox created

> kubectl get pods -o wide
NAME            READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
busybox-mn98w   1/1     Running   0          84s   10.0.0.58    tomoyafujita   <none>           <none>
busybox-z2mbw   1/1     Running   0          84s   10.0.1.121   edgemaster     <none>           <none>

> kubectl exec --stdin --tty busybox-mn98w -- /bin/sh
/ #
/ # ping 10.0.1.121
PING 10.0.1.121 (10.0.1.121): 56 data bytes
64 bytes from 10.0.1.121: seq=0 ttl=63 time=1.326 ms
64 bytes from 10.0.1.121: seq=1 ttl=63 time=1.620 ms
64 bytes from 10.0.1.121: seq=2 ttl=63 time=1.341 ms
64 bytes from 10.0.1.121: seq=3 ttl=63 time=1.685 ms
^C
--- 10.0.1.121 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 1.326/1.493/1.685 ms
/ # exit
> kubectl exec --stdin --tty busybox-z2mbw -- /bin/sh
/ #
/ # ping 10.0.0.58
PING 10.0.0.58 (10.0.0.58): 56 data bytes
64 bytes from 10.0.0.58: seq=0 ttl=63 time=0.728 ms
64 bytes from 10.0.0.58: seq=1 ttl=63 time=1.178 ms
64 bytes from 10.0.0.58: seq=2 ttl=63 time=0.635 ms
64 bytes from 10.0.0.58: seq=3 ttl=63 time=1.152 ms
^C
--- 10.0.0.58 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.635/0.923/1.178 ms

Finally we can confirm the cross-communication via Cilium between busybox containers are just fine!!!

If you want to know more technical insights and development, see and subscribe KubeEdge EdgeCore supports Cilium CNI for more details.

1.17 What's New​

• Support Edge Pods Using InClusterConfig to Access Kube-APIServer
• Mapper Supports Video Streaming Data Reporting
• Support Auto-Restarting for Edge Modules
• Introduce keadm ctl Command to Support Pods Query and Restart at Edge
• Keadm Enhancements
• Add MySQL to Mapper Framework
• Upgrade Kubernetes Dependency to v1.28.6

Release Highlights​

Support Edge Pods Using InClusterConfig to Access Kube-APIServer​

The InClusterConfig mechanism enables cloud pods to directly access the Kubernetes API server. With this release, KubeEdge now supports edge pods using the InClusterConfig mechanism to access the Kube-APIServer directly, even when the edge and cloud are in different network environments.

Refer to the link for more details. (#5524, #5541)

Mapper Supports Video Streaming Data Reporting​

Previously, Mappers could only process structured device data. In v1.17, video stream data processing features have been added to the Mapper-Framework.

• Edge Camera Device Management

  v1.17 provides a built-in Mapper based on the Onvif protocol, which can manage Onvif network camera devices into the KubeEdge cluster and obtain the camera's authentication file and RTSP video stream.

• Video Stream Data Processing

  Video stream data processing capabilities have been introduced to the Mapper-Framework data plane. The video stream reported by edge camera devices can be saved as frame files or video files.

Refer to the link for more details. (#5448, #5514, mappers-go/#127)

Support Auto-Restarting for Edge Modules​

EdgeCore modules could previously fail to start due to non-configurable and recoverable matters like process start order issues. In v1.17, the BeeHive framework has been improved to support automatically restarting modules. Users can now configure EdgeCore modules to automatically restart instead of restarting the entire component.

Refer to the link for more details. (#5509, #5513)

Introduce keadm ctl Command to Support Pods Query and Restart at Edge​

The new keadm ctl command has been introduced in v1.17, allowing users to query and restart pods on edge nodes when they are offline:

• Query: keadm ctl get pod [flags]
• Restart: keadm ctl restart pod [flags]

Refer to the link for more details. (#5504)

Keadm Enhancements​

Several enhancements were made to the keadm installation tool:

• Refactored the keadm init command
• Changed the command keadm generate to keadm manifest
• Added image-repository flag to keadm join to support customization
• Split the keadm reset command into keadm reset cloud and keadm reset edge.

Refer to the link for more details. (#5317)

Add MySQL to Mapper Framework​

The Mapper Framework data plane now includes MySQL database support in its pushMethod. When using MySQL, basic configuration parameters for the MySQL client need to be added in the DeviceInstance.

Refer to the link for more details. (#5376)

Upgrade Kubernetes Dependency to v1.28.6​

The vendored Kubernetes version has been upgraded to v1.28.6, users are now able to use the latest features on both the cloud and edge side.

Refer to the link for more details. (#5412)

Important Steps before Upgrading​

• To use the InClusterConfig feature for edge pods, you need to enable the metaServer and dynamicController switches, and set featureGates.requireAuthorization=true in the CloudCore and EdgeCore configuration files.

• To use the Auto-Restarting for Edge Modules feature, you must enable the moduleRestart feature gate in EdgeCore.

Download the v1.17.0 release from the release page and upgrade today to take advantage of these new capabilities!

v1.16 What's New​

• Support Cloud and Edge Components Upgrade
• Alpha Implementation of Images PrePull on Edge Nodes
• Support Installing Windows-based Edge Nodes with Keadm
• Add Compatibility Tests for Multiple Runtimes
• Support More Deployment Fields to the EdgeApplication Overrides
• Support Mapper Upgrade
• Integrate Redis and TDengine Database in DMI Data Plane
• New USB Camera Mapper
• Keadm’s Enhancement
• Upgrade Kubernetes Dependency to v1.27.7

Release Highlights​

Support Cloud and Edge Components Upgrade​

The Cloud side and Edge side Upgrade capability is comprehensively enhanced in v1.16. Users can upgrade the cloud side components with Keadm tool, and upgrade edge nodes with the API through Kubernetes API-Server.

• Cloud upgrade

  Keadm supports the Cloud Upgrade command, and users can easily upgrade cloud components.

  Example:

  keadm upgrade cloud --advertise-address={advertise-address} --kubeedge-version=v1.16.0
  

• Edge upgrade

  In KubeEdge v1.16, the node upgrade API was implemented. Users can remotely upgrade edge nodes in batches. The cloud-edge task architecture handles upgrade task flow and supports unified timeout processing, concurrency control, and subtask management, among other capabilities.

  Upgrade API Example:

  apiVersion: operations.kubeedge.io/v1alpha1
  kind: NodeUpgradeJob
  metadata:
    name: upgrade-example
    labels:
      description: upgrade-label
  spec:
    version: "v1.16.0"
    checkItems:
      - "cpu"
      - "mem"
      - "disk"
    failureTolerate: "0.3"
    concurrency: 2
    timeoutSeconds: 180
    labelSelector:
      matchLabels:
        "node-role.kubernetes.io/edge": ""
        node-role.kubernetes.io/agent: ""
  

• KubeEdge version compatibility testing

  KubeEdge v1.16 provides KubeEdge version compatibility testing, which avoids problems caused by incompatible cloud-edge versions during the upgrading process.

Refer to the link for more details. (#5330, #5229, #5289)

Alpha Implementation of Images PrePull on Edge Nodes​

In scenarios with unstable network or limited edge bandwidth, deploying or updating edge applications often results in high failure rates or reduced efficiency, especially with large-scale edge nodes.

Images PrePull feature has been introduced in v1.16. Users can perform batch images prepull on large-scale edge nodes with ImagePrePullJob API when the network is stable, to improve the success rate and efficiency of batch edge applications deploying and updating.

ImagePrePull API Example:

apiVersion: operations.kubeedge.io/v1alpha1
kind: ImagePrePullJob
metadata:
  name: imageprepull-example
  labels:
    description:ImagePrePullLabel
spec:
  imagePrePullTemplate：
    images:
      - image1
      - image2
    nodes：
      - edgenode1
      - edgenode2
    checkItems:
      - "disk"
    failureTolerate: "0.3"
    concurrency: 2
    timeoutSeconds: 180
    retryTimes: 1

Refer to the link for more details. (#5310, #5331)

Support Installing Windows-based Edge Nodes with Keadm​

KubeEdge has supported the edge node running on Windows Server 2019 in v1.15, extending KubeEdge to the Windows ecosystem and expanding its use cases and ecosystem.

In this release, Windows-based Edge Nodes can be installed and registered to cloud with the installation tool Keadm, providing convenience for the application of KubeEdge in Windows OS.

Refer to the link for more details. (#4968)

Add Compatibility Tests for Multiple Runtimes​

The e2e test of KubeEdge v1.16 has integrated compatibility tests for multiple container runtimes. Currently, four container runtime compatibility tests have been added, including containerd, docker, cri-o, and isulad.

Refer to the link for more details.(#5321)

Support More Deployment Fields to the EdgeApplication Overrides​

In previous versions, only replicas and image of the EdgeApplication could be overridden. In this release, we support overriding more Deployment fields: env, command, args and resources.

Refer to the link for more details.(#5038)

Support Mapper Upgrade​

Build mapper upgrade framework. Users can upgrade the mapper by changing the referenced mapper-framework package version.

• Mapper-framework code decouple

  The code in mapper-framework was decoupled into user-layer code and business-layer code, and create the kubeedge/mapper-framework repo to store the business layer code.

• Mapper upgrade framework

  Update the way mapper-framework generates mapper projects. The current execution script will only generate user-level code through dependent references. When the mapper project needs to be upgraded, it can be directly made by changing the version of mapper-framework package.

Refer to the link for more details.(#5308, #5326)

Integrate Redis and TDengine Database in DMI Data Plane​

Integrate redis and tdengine database in DMI data plane. The mapper project generated by mapper-framework has built-in ability to push data to redis and tdengine database. Users can push data directly through configuring device instance files.

Database Field Definition:

type DBMethodRedis struct {
    // RedisClientConfig of redis database
    // +optional
    RedisClientConfig *RedisClientConfig `json:"redisClientConfig,omitempty"`
}
type RedisClientConfig struct {
    // Addr of Redis database
    // +optional
    Addr string `json:"addr,omitempty"`
    // Db of Redis database
    // +optional
    DB int `json:"db,omitempty"`
    // Poolsize of Redis database
    // +optional
    Poolsize int `json:"poo lsize,omitempty"`
    // MinIdleConns of Redis database
    // +optional
    MinIdleConns int `json:"minIdleConns,omitempty"`
}

type DBMethodTDEngine struct {
    // tdengineClientConfig of tdengine database
    // +optional
    TDEngineClientConfig *TDEngineClientConfig `json:"TDEngineClientConfig,omitempty"`
}
type TDEngineClientConfig struct {
    // addr of tdEngine database
    // +optional
    Addr string `json:"addr,omitempty"`
    // dbname of tdEngine database
    // +optional
    DBName string `json:"dbName,omitempty"`
}

Refer to the link for more details.(#5064)

New USB Camera Mapper​

Based on the mapper and dmi framework in KubeEdge v1.15.0, a mapper for USB cameras has been developed, which supports data push to Influxdb, mqtt, and http. It has been successfully applied in practice.

Refer to the link for more details.(#122)

Keadm’s Enhancement​

• When using Keadm join in kubeEdge v1.16, it supports the selection of communication protocols for edge nodes and cloud center nodes. The cloud edge communication protocol is configured through the parameter --hub-protocol, and currently supports two communication protocols: websocket and quic.

  note

  When the --hub-protocol parameter is configured as quic, it is necessary to set the port of the parameter --cloudcore-ipport to 10001 and modify configmap in cloudcore to open the quic protocol.

  Refer to the link for more details.(#5156)

• In KubeEdge v1.16, it is already supported for Keadm to complete edgecore deployment through Keadm join without installing the CNI plugin, decoupling the deployment of edge nodes from the CNI plugin. At the same time, this feature has been synchronized to v1.12 and later versions.

  note

  If the application deployed on edge nodes needs to use container networks, it is still necessary to install the CNI plugin after deploying edgecore.

  Refer to the link for more details.(#5196)

Upgrade Kubernetes Dependency to v1.27.7​

Upgrade the vendered kubernetes version to v1.27.7, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#5121)

Important Steps before Upgrading​

• Now we use DaemonSet to manage the mqtt broker mosquitto. You need to consider whether to use the static pod managed mqtt broker in the edge node or use the DaemonSet managed mqtt broker in the cloud, they cannot coexist and there will be port conflicts. You can read the guide For edge node low version compatibility in #5233.

• In this release, the flag with-mqtt will be set to deprecated and default to false, but will not be removed. After v1.18, the code related to static pod management will be removed in the edge, and the flag with-mqtt no longer supported.

v1.15 What's New​

• Support Windows-based Edge Nodes

• New v1beta1 version of Device API

• Support Alpha version of DMI DatePlane and Mapper-Framework

• Support Kubernetes native Static Pod on Edge Nodes

• Support more Kubernetes Native Plugin Running on Edge Node

• Upgrade Kubernetes Dependency to v1.26.7

Release Highlights​

Support Windows-based Edge Nodes​

Edge computing involves various types of devices, including sensors, cameras, and industrial control devices, some of which may run on the Windows OS. In order to support these devices and use cases, supporting Windows Server nodes is necessary for KubeEdge.

In this release, KubeEdge supports the edge node running on Windows Server 2019, and supports Windows container running on edge node, thereby extending KubeEdge to the Windows ecosystem and expanding its use cases and ecosystem.

Refer to the link for more details. (#4914, #4967)

New v1beta1 version of Device API​

The device API is updated from v1alpha2 to v1beta1, in v1beta1 API updates include:

• The built-in protocols include Modbus, Opc-UA and Bluetooth are removed in device instance, and the built-in mappers for these proytocols still exists and will be maintained and updated to latest version.

• Users must define the protocol config through CustomizedValue in ProtocolConfig.

• DMI date plane related fields are added, users can config the collection and reporting frequency of device data, and the destination to which(such as database, httpserver) data is pushed.

• Controls whether to report device data to cloud.

Refer to the link for more details. (#4983)

Support Alpha version of DMI DatePlane and Mapper-Framework​

Alpha version of DMI date plane is supported, DMI date plane is mainly implemented in mapper, providing interface for pushing data, pulling data, and storing data in database.

To make writing mapper easier, a mapper development framework subproject Mapper-Framework is provided in this release. Mapper-Framework provides mapper runtime libs and tools for scaffolding and code generation to bootstrap a new mapper project. Users only need to run a command make generate to generate a mapper project, then add protocol related code to mapper.

Refer to the link for more details. (#5023)

Support Kubernetes native Static Pod on Edge Nodes​

Kubernetes native Static Pod is supported on edge node in this release. Users can create pods on edge nodes by place pod manifests in /etc/kubeedge/manifests, same as that on the Kubernetes node.

Refer to the link for more details. (#4825)

Support more Kubernetes Native Plugin Running on Edge Node​

Kubernetes non-resource kind request /version is supported from edge node, users now can do /version requests in edge node from metaserver. In addition, it can easily support other non-resource kind of requests like /healthz in edge node with the current framework. Many kubernetes plugins like cilium/calico which depend on these non-resource kind of requests, now can run on edge nodes.

Refer to the link for more details. (#4904)

Upgrade Kubernetes Dependency to v1.26.7​

Upgrade the vendered kubernetes version to v1.26.7, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#4929)

Important Steps before Upgrading​

• In KubeEdge v1.15, new v1beta1 version of device API is incompatible with earlier versions of v1alpha1, users need to update the device API yamls to v1bata1 if you want to use v1.15.

• In KubeEdge v1.15, users need to upgrade the containerd to v1.6.0 or later. Containerd minor version 1.5 and older will not be supported in KubeEdge v1.15.
  Ref: https://kubernetes.io/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/#cri-api-removal

• In KubeEdge v1.14, EdgeCore has removed the dockeshim support, so users can only use remote type runtime, and uses containerd runtime by default. If you want to use docker runtime in v1.15, you also need to first set edged.containerRuntime=remote and corresponding docker configuration like RemoteRuntimeEndpoint and RemoteImageEndpoint in EdgeCore, then install the cri-dockerd tools as docs below: https://github.com/kubeedge/kubeedge/issues/4843

v1.14 What's New​

• Support Authentication and Authorization for Kube-API Endpoint for Applications On Edge Nodes

• Support Cluster Scope Resource Reliable Delivery to Edge Node

• Upgrade Kubernetes Dependency to v1.24.14

• Support Kubectl Attach to Container Running on Edge Node

• Alpha version of KubeEdge Dashboard

Release Highlights​

Support Authentication and Authorization for Kube-API Endpoint for Applications On Edge Nodes​

The Kube-API endpoint for edge applications is implemented through MetaServer in edegcore. However, in previous versions, the authentication and authorization of Kube-API endpoint are performed in the cloud, which prevents authentication and authorization especially in offline scenarios on the edge node.

In this release, the authentication and authorization functionalities are implemented within the MetaServer at edge, which allows for limiting the access permissions of edge applications when accessing Kube-API endpoint at edge.

Refer to the link for more details. (#4802)

Support Cluster Scope Resource Reliable Delivery to Edge Node​

The cluster scope resource can guarantee deliver to the edge side reliably since this release, especially include using list-watch global resources, the cluster scope resource can be delivered to the edge side reliably, and the edge applications can work normally.

Refer to the link for more details. (#4758)

Upgrade Kubernetes Dependency to v1.24.14​

Upgrade the vendered kubernetes version to v1.24.14, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#4789)

Support Kubectl Attach to Container Running on Edge Node​

KubeEdge already support kubectl logs/exe command, kubectl attach is supported in this release. kubectl attach command can attach to a running container at edge node. Users can execute these commands in the cloud and no need to operate on the edge nodes.

Refer to the link for more details. (#4734)

Alpha version of KubeEdge Dashboard​

KubeEdge dashboard provides a graphical user interface (GUI) for managing and monitoring your KubeEdge clusters. It allows users to manage edge applications running in the cluster and troubleshoot them.

Refer to the link for more details. (https://github.com/kubeedge/dashboard)

Important Steps before Upgrading​

• On KubeEdge v1.14, EdgeCore has removed the dockeshim support, so users can only use remote type runtime, and uses containerd runtime by default. If you want to use docker runtime, you must first set edged.containerRuntime=remote and corresponding docker configuration like RemoteRuntimeEndpoint and RemoteImageEndpoint in EdgeCore, then install the cri-dockerd tools as docs below: https://github.com/kubeedge/kubeedge/issues/4843

Why Is SLSA 3 Compliance Important to KubeEdge​

Software supply chain integrity attacks (unauthorized modification of software packages) have been increasing in the past three years. After KubeEdge reaches SLSA 3, the E2E security (from source code build to release) is hardened, preventing malicious tampering of binary and container image artifacts. Thanks to SLSA, we can enhance the integrity of software artifacts, against tampering or any type of unauthorized modification of the software or software package, and enable software to defend against common supply chain attacks.

About SLSA​

Proposed by Google, SLSA (Supply Chain Levels for Software Artifacts) is a security framework and a check-list of standards and controls to prevent tampering, improve software supply chain integrity, and secure software packages and infrastructure in projects, businesses or enterprises. It is not a single tool, but a step-by-step outline to prevent artifacts from being tampered with and tampered artifacts from being used, and at the higher levels, to strengthen the platforms that make up a supply chain. Manufacturers follow the SLSA guidelines to safeguard their software, and users make decisions based on the security status of software packages.

SLSA is currently in alpha, and related definitions are subject to change.

The following figure describes the known attack points in software supply chains. For details, see https://slsa.dev/.

SLSA introduces many new tools and concepts, such as:

• Artifact: any artifact file generated by a build pipeline, such as container images, language packages, and compiled binary files.
• Provenance: built metadata, including the build steps, build sources, and dependencies.
• Digest: the result of a cryptographic hash function. The hash function generates a fixed-size value that uniquely identifies an artifact, such as an SHA-256 hash of a container image.
• Attestation: an encrypted and signed file that records the source of software artifacts.
• Immutable references: an identifier that is guaranteed to always point to the same, immutable artifact, such as a specific container image or language package.
• Build integrity: verifies the output integrity of the build pipeline.

How KubeEdge Reaches SLSA 3​

The SLSA requirements are arranged into four levels. The higher the level, the higher the requirements. For details, see https://slsa.dev/spec/v0.1/requirements.

When the third-party security audit report was released in July 2022, KubeEdge did not meet SLSA 3 requirements in terms of SLSA provenance. After continuous hardening by SIG-Security, KubeEdge finally achieved SLSA 3 in January 2023. The following table lists the compliance status of KubeEdge in Source, Build, Provenance, and Common. ("Y" indicates compliance, while blank indicates non-compliance.)

SLSA requirements and compliance

This section describes how KubeEdge meets the build and provenance requirements of SLSA 3. The build/provenance requirements and corresponding KubeEdge solutions are as follows:

Build requirements:

a) Scripted build: All build steps were fully defined in some sort of "build script".

b) Build service: All build steps ran using some build services, not on a developer's workstation. Build services include GitHub Actions and those provided by third-party cloud platforms.

c) Build as code: The build definition and configuration executed by the build service is verifiably derived from text file definitions stored in a version control system.

d) Ephemeral environment: The build service ensured that the build steps ran in an ephemeral environment, such as a container or VM, provisioned solely for this build, and not reused from a prior build.

e) Isolated: The build service ensured that the build steps ran in an isolated environment free of influence from other build instances, whether prior or concurrent.

f) Parameterless: The build output cannot be affected by user parameters other than the build entry point and the top-level source location. In other words, the build is fully defined through the build script and nothing else.

g) Hermetic: All transitive build steps, sources, and dependencies were fully declared up front with immutable references, and the build steps ran with no network access.

Solution:

All KubeEdge build processes are automatically executed by scripts on GitHub. As a build service (with definition files and configuration files stored in the .github/workflows directory), GitHub Actions ensure that build steps are traceable and verifiable, the build environment is ephemeral and isolated, and the build parameters and dependencies cannot be tampered with.

Provenance requirements:

a) Available: The provenance is available to the consumer in a format that the consumer accepts. The format SHOULD be in-toto SLSA Provenance, but another format MAY be used if both producer and consumer agree and it meets all the other requirements.

b) Authenticated: The provenance's authenticity and integrity can be verified by the consumer. This SHOULD be through a digital signature from a private key accessible only to the service generating the provenance.

c) Service generated: The data in the provenance MUST be obtained from the build service.

d) Non-falsifiable: Provenance cannot be falsified by the build service's users.

e) Dependencies complete: Provenance records all build dependencies that were available while running the build steps. This includes the initial state of the machine, VM, or container of the build worker.

Solution:

slsa-github-generator, SLSA's official GitHub build project, is integrated into the artifacts (including binary files and container images) released in KubeEdge to achieve SLSA 3.

The KubeEdge version release process (.github/workflows/release.yml) integrates slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml and slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml to ensure that the process of building and releasing artifacts (including binary files and container images) meets the SLSA 3 requirements.

For details about slsa-github-generator, see https://github.com/slsa-framework/slsa-github-generator.

About Provenance​

Provenance is the build metadata, a proof of the software build and release execution process and can be verified. It includes build steps, build sources, and dependencies, such as source code repositories, code branches, and configuration files. Provenance content of SLSA 3 is authentic, tamper-proof, and will not be changed by project maintainers. The provenance file (named multiple.intoto.jsonl) of binary artifacts is released with the release software package. The provenance file of a container image is uploaded to the KubeEdge Docker Hub public repository together with the image. For details about the Provenance formats, see https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/generic/README.md#provenance-format.

How to Verify That KubeEdge Release Artifacts Meet SLSA 3​

For details, see https://github.com/kubeedge/kubeedge/pull/4285.

The following is an example:

$ COSIGN_EXPERIMENTAL=1 cosign verify-attestation --type slsaprovenance --policy policy.cue kubeedge/cloudcore:v1.13.0

{
    "_type": "https://in-toto.io/Statement/v0.1",
    "predicateType": "https://slsa.dev/provenance/v0.2",
    "subject": [{
            "name": "index.docker.io/kubeedge/cloudcore",
            "digest": {
                "sha256": "825642e63ab5b924e2fa0661cd14d544d0be151c4bdba6f3f42796c977fbe211"
            }
        }
    ],
    "predicate": {
        "builder": {
            "id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.4.0"
        },
        "buildType": "https://github.com/slsa-framework/slsa-github-generator/container@v1",
        "invocation": {
            "configSource": {
                "uri": "git+https://github.com/kubeedge/kubeedge@refs/tags/v1.13.0",
                "digest": {
                    "sha1": "ee357a0d5341241143e88d45df99fde865c987de"
                },
                "entryPoint": ".github/workflows/release.yml"
            },
...

Signing and Working Principles of slsa-github-generator​

OpenID Connect (OIDC) is used to prove the identity of a workflow to an external service (Sigstore). OIDC is a standard used across the web by identity providers to authenticate users for third parties. GitHub now supports OIDC in its workflows. Each time a workflow runs, a runner can create a unique JWT token from GitHub's OIDC provider. The token contains verifiable information about the workflow identity, including the caller repository, commit hash, trigger, and current (reusable) workflow path and reference.

With OIDC, the workflow proves its identity to Sigstore's Fulcio root certificate authority (an external authentication service). Fulcio signs a short-term certificate to prove the temporary signature key (generated in the runner) and binds the key to the workload identity. The signing record is stored in Sigstore's transparency log Rekor. A user can use the signed certificate as a trusted anchor to verify that the provenance was authenticated and cannot be falsified; it must have been created in a trusted builder. The process is as follows:

In 2022, slsa-github-generator was honored by the Sigstore community as Best User Adopter.

Summary​

SLSA plays an important role in the software supply chain security of KubeEdge. Based on the Sigstore capabilities, the entire software supply chain process of KubeEdge, from source code to release artifacts, is signed and verified to improve the security.

References:

[1] KubeEdge blog: https://kubeedge.io/en/blog/reach-slsa-l3/

[2] The third-party security audit of KubeEdge: https://github.com/kubeedge/community/blob/master/sig-security/sig-security-audit/KubeEdge-security-audit-2022.pdf

[3] SLSA website: https://slsa.dev/

[4] Sigstore website: https://www.sigstore.dev/

[5] SLSA Generator: https://github.com/slsa-framework/slsa-github-generator

[6] SLSA3 generic integration guide: https://slsa.dev/blog/2022/08/slsa-github-workflows-generic-ga

[7] Improving software supply chain security with tamper-proof builds: https://security.googleblog.com/2022/04/improving-software-supply-chain.html

[8] Sigstore November Roundup: https://blog.sigstore.dev/sigstore-november-roundup-8a852cec10fc/

v1.13 What's New​

• Performance Improvement

• Security Improvement

• Upgrade Kubernetes Dependency to v1.23.15

• Modbus Mapper based on DMI

• Support Rolling Upgrade for Edge Nodes from Cloud

• Test Runner for conformance test

• EdgeMesh: Added configurable field TunnelLimitConfig to edge-tunnel module

Performance Improvement​

• CloudCore memory usage is reduced by 40%, through unified generic Informer and reduce unnecessary cache. (#4375, #4377)

• List-watch dynamicController processing optimization, each watcher has a separate channel and goroutine processing to improve processing efficiency (#4506)

• Added list-watch synchronization mechanism between cloud and edge and add dynamicController watch gc mechanism (#4484)

• Removed 10s hard delay when offline nodes turn online (#4490)

• Added prometheus monitor server and a metric connected_nodes to cloudHub. This metric tallies the number of connected nodes each cloudhub instance (#3646)

• Added pprof for visualization and analysis of profiling data (#3646)

• CloudCore configuration is now automatically adjusted according to nodeLimit to adapt to the number of nodes of different scales (#4376)

Security Improvement​

• KubeEdge is proud to announce that we are digitally signing all release artifacts (including binary artifacts and container images). Signing artifacts provides end users a chance to verify the integrity of the downloaded resource. It allows to mitigate man-in-the-middle attacks directly on the client side and therefore ensures the trustfulness of the remote serving the artifacts. By doing this, we reached the SLSA security assessment level L3 (#4285)

• Remove the token field in the edge node configuration file edgecore.yaml to eliminate the risk of edge information leakage (#4488)

Upgrade Kubernetes Dependency to v1.23.15​

Upgrade the vendered kubernetes version to v1.23.15, users are now able to use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#4509)

Modbus Mapper based on DMI​

Modbus Device Mapper based on DMI is provided, which is used to access Modbus protocol devices and uses DMI to synchronize the management plane messages of devices with edgecore.

Refer to the link for more details. (mappers-go#79)

Support Rolling Upgrade for Edge Nodes from Cloud​

Users now able to trigger rolling upgrade for edge nodes from cloud, and specify number of concurrent upgrade nodes with nodeupgradejob.spec.concurrency. The default Concurrency value is 1, which means upgrade edge nodes one by one.

Refer to the link for more details. (#4476)

Test Runner for conformance test​

KubeEdge has provided the runner of the conformance test, which contains the scripts and related files of the conformance test.

Refer to the link for more details. (#4411)

EdgeMesh: Added configurable field TunnelLimitConfig to edge-tunnel module​

The tunnel stream of the edge-tunnel module is used to manage the data stream state of the tunnel. Users can obtain a stable and configurable tunnel stream to ensure the reliability of user application traffic forwarding.

Users can configure the cache size of tunnel stream according to TunnelLimitConfig to support larger application relay traffic.

Refer to the link for more details. (#399)

Cancel the restrictions on the relay to ensure the stability of the user's streaming application or long link application.

Refer to the link for more details. (#400)

Important Steps before Upgrading​

• EdgeCore now uses containerd runtime by default on KubeEdge v1.13. If you want to use docker runtime, you must set edged.containerRuntime=docker and corresponding docker configuration like DockerEndpoint, RemoteRuntimeEndpoint and RemoteImageEndpoint in EdgeCore.

What's New in KubeEdge v1.12​

• Alpha Implementation of Next-Gen Cloud Native Device Management Interface (DMI)

• Next-Gen Edged Graduates to GA: Suitable for More Scenarios

• Introducing High-Availability Mode for EdgeMesh

• Support Edge Node Upgrade from the Cloud

• Support Authorization for Edge Kube-API Endpoint

• New GigE Mapper

Alpha Implementation of Next-Gen Cloud Native Device Management Interface (DMI)​

DMI makes KubeEdge's IoT device management more pluggable and modular in a cloud-native way, covering Device Lifecycle Management, Device Operation, and Device Data Management.

• Device Lifecycle Management: Simplifies IoT device lifecycle management, making it as easy as managing a pod.

• Device Operation: Provides the ability to operate devices through the Kubernetes API.

• Device Data Management: Separates device data management from device management, allowing data to be consumed by local applications or synchronized to the cloud through a special tunnel.

Next-Gen Edged Graduates to GA: Suitable for More Scenarios​

The new version of the lightweight Edged engine, optimized from Kubelet and integrated into EdgeCore, has graduated to General Availability (GA) in this release. The new Edged will continue to communicate with the cloud through a reliable transmission tunnel, making it suitable for a wider range of scenarios.

Introducing High-Availability Mode for EdgeMesh​

KubeEdge v1.12 introduces a high-availability mode for EdgeMesh. Unlike the previous centralized relay mode, the EdgeMesh HA mode can set up multiple relay nodes. When some relay nodes fail, other relay nodes can continue to provide relay services, avoiding single points of failure and improving system stability.

Support Edge Node Upgrade from the Cloud​

KubeEdge v1.12 introduces the NodeUpgradeJob v1alpha1 API to upgrade edge nodes from the cloud. With this API and its associated controller, users can upgrade selected edge nodes from the cloud and roll back to the original version if the upgrade fails.

Support Authorization for Edge Kube-API Endpoint​

Authorization for the Edge Kube-API Endpoint is now available in KubeEdge v1.12. Third-party plugins and applications that depend on Kubernetes APIs on edge nodes must use a bearer token to communicate with the kube-apiserver via the HTTPS server in MetaServer.

New GigE Mapper​

KubeEdge v1.12 includes a new GigE Device Mapper with a Golang implementation, which is used to access GigE Vision protocol cameras.

Important Steps Before Upgrading​

• If you want to upgrade KubeEdge to v1.12, the configuration file in EdgeCore has been upgraded to v1alpha2. You must modify your configuration file for Edged in EdgeCore to adapt to the new Edged.

• If you want to use authorization for the Edge Kube-API Endpoint, please enable the RequireAuthorization feature through the feature gate in both CloudCore and EdgeCore. If the RequireAuthorization feature is enabled, MetaServer will only serve HTTPS requests.

• If you want to upgrade EdgeMesh to v1.12, you do not need to deploy the existing EdgeMesh-server, but you need to configure relayNodes.

• If you want to run EdgeMesh v1.12 on KubeEdge v1.12 and use HTTPS requests to communicate with KubeEdge, you must set kubeAPIConfig.metaServer.security.enable=true.

KubeEdge v1.12 brings exciting new features and improvements to the edge computing ecosystem. We invite you to explore the release and provide feedback to the community. Happy edge computing!

The population of KubeEdge brings in community interests in the scalability and scale of KubeEdge. Now, Kubernetes clusters powered by KubeEdge, as fully tested, can stably support 100,000 concurrent edge nodes and manage more than one million pods. This report introduces the metrics used in the test, the test procedure, and the method to connect to an ocean of edge nodes.

Background​

Fast growing technologies, such as 5G networks, industrial Internet, and AI, are giving edge computing an important role in driving digital transformation. Cities, transportation, healthcare, manufacturing, and many other fields are becoming smart thanks to edge computing. According to Gartner, by 2023, the number of intelligent edge devices may be more than 20 times that of traditional IT devices. By 2028, the embedding of sensors, storage, computing, and advanced AI functions in edge devices will grow steadily. IoT devices are of various types and in large quantities. The increasing connected IoT devices are challenging management and O&M.

At the same time, users in the KubeEdge community are expecting large-scale edge deployment. There are already some successful use cases for KubeEdge. In unmanned toll stations across China, there are nearly 100,000 edge nodes and more than 500,000 edge applications in this project, and the numbers keep growing. Another case is a vehicle-cloud collaboration platform, the industry-first cloud-edge-device system. It enables fast software upgrade and iteration for software-defined vehicles. On this platform, each vehicle is connected as an edge node, and the number of edge nodes will reach millions.

Introduction to KubeEdge​

KubeEdge is the industry's first cloud native edge computing framework designed for edge-cloud collaboration. Complementing Kubernetes for container orchestration and scheduling, KubeEdge allows applications, resources, data, and devices to collaborate between edges and the cloud. Devices, edges, and the cloud are now fully connected in edge computing.

In the KubeEdge architecture, the cloud is a unified control plane, which includes native Kubernetes management components and KubeEdge-developed CloudCore components. It listens to cloud resource changes and provides reliable, efficient cloud-edge messaging. At the edge side lie the EdgeCore components, including Edged, MetaManager, and EdgeHub. They receive messages from the cloud and manage the lifecycle of containers. The device mapper and event bus are responsible for device access.

Based on the Kubernetes control plane, KubeEdge allows nodes to be deployed more remotely and thereby extends edge-cloud collaboration. Kubernetes supports 5,000 nodes and 150,000 pods, which are far from enough for edge computing in Internet of Everything (IoE). The access of a large number of edge devices demands a scalable, centralized edge computing platform. To help users cost less and manage more in an easier way, KubeEdge, fully compatible with Kubernetes, optimizes the cloud-edge messaging and provides access support for mass edge nodes.

SLIs/SLOs​

Scalability and performance are important features of Kubernetes clusters. Before performing the large-scale performance test, we need to define the measurement metrics. The Kubernetes community defines the following SLIs (Service Level Indicators) and SLOs (Service Level Objectives) to measure the cluster service quality.

1. API Call Latency

2. Pod Startup Latency

The community also defines indicators such as in-cluster network programming latency (latency for Service updates or changes in ready pods to be reflected to iptables/IPVS rules), in-cluster network latency, DNS programming latency (latency for Service updates or changes in ready pods to be reflected to the DNS server), and DNS latency. These indicators have not yet been quantified. This test was conducted to satisfy all SLIs/SLOs in the official state.

Kubernetes Scalability Dimensions and Thresholds​

Kubernetes scalability does not just mean the number of nodes (Scalability != #Nodes). Other dimensions include the number of namespaces, pods, Services, secrets, and ConfigMaps. Configurations that Kubernetes supports create the Scalability Envelope (which keeps evolving):

Obviously, it is impossible for a Kubernetes cluster to expand resource objects without limitation while satisfying SLIs/SLOs. Therefore, the industry defines the upper limits of Kubernetes resource objects.

1. Pods/node 30
2. Backends <= 50k & Services <= 10k & Backends/service <= 250
3. Pod churn 20/s
4. Secret & configmap/node 30
5. Namespaces <= 10k & Pods <= 150k & Pods/namespace <= 3k
6. ​    …..

Dimensions are sometimes not independent. As you move farther along one dimension, your cross-section wrt other dimensions gets smaller. For example, if 5000 nodes are expanded to 10,000 nodes, the specifications of other dimensions will be affected. A heavy workload is required if all scenarios are tested. In this test, we focus on the typical scenarios. We manage to host 100k edge nodes and 1000k pods in a single cluster while satisfying the SLIs/SLOs.

Test Tools​

ClusterLoader2​

ClusterLoader2 is an open source Kubernetes cluster performance test tool. It can test the Kubernetes SLIs/SLOs to check whether the cluster meets the service quality standards. It also visualizes data for locating cluster problems and optimizing cluster performance. After the test, users get a performance report with detailed test results.

Clusterloader2 performance metrics:

• APIResponsivenessPrometheusSimple
• APIResponsivenessPrometheus
• CPUProfile
• EtcdMetrics
• MemoryProfile
• MetricsForE2E
• PodStartupLatency
• ResourceUsageSummary
• SchedulingMetrics
• SchedulingThroughput
• WaitForControlledPodsRunning
• WaitForRunningPods

Edgemark​

Edgemark is a performance test tool similar to Kubemark. It simulates deploying KubeEdge edge nodes in the KubeEdge cluster scalability test to build ultra-large Kubernetes clusters, powered by KubeEdge, with limited resources. The objective is to expose the cluster control plane downsides that occur only in large-scale deployments. The following figure illustrates the Edgemark deployment:

• K8s master: the master node of the Kubernetes cluster
• Edgemark master: the master node of the simulated Kubernetes cluster
• CloudCore: the KubeEdge cloud management component, which is responsible for edge node access
• hollow pod: a pod started in the actual cluster. It registers with the Edgemark master as a virtual edge node by starting Edgemark in it. The Edgemark master can schedule pods to this virtual edge node.
• hollow edgeNode: a virtual node in the simulated cluster, registered from a hollow pod

Cluster Deployment Scheme for the Test​

The Kubernetes control plane is deployed with one master node. The etcd, kube-apiserver, kube-scheduler, and kube-controller are deployed as single-instance. The KubeEdge control plane is deployed with five CloudCore instances and connects to the kube-apiserver through the IP address of the master node. Hollow EdgeNodes are exposed by a load balancer and randomly connect to a CloudCore instance based on the round-robin policy of the load balancer.

Test Environment Information​

Control Plane OS Version

CentOS 7.9 64bit 3.10.0-1160.15.2.el7.x86_64

Kubernetes Version

Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"

KubeEdge Version

KubeEdge v1.11.0-alpha.0

Master Node Configurations

• CPU

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 106
Model name:            Intel(R) Xeon(R) Platinum 8378A CPU @ 3.00GHz
Stepping:              6
CPU MHz:               2999.998

• MEMORY

Total online memory:   256G

• ETCD DISK

Type:   SAS_SSD
Size:   300GB

CloudCore Node Configurations

• CPU

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                12
On-line CPU(s) list:   0-11
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 106
Model name:            Intel(R) Xeon(R) Platinum 8378A CPU @ 3.00GHz
Stepping:              6
CPU MHz:               2999.998

• MEMORY

Total online memory:   48G

Component Parameter Configurations​

1. kube-apiserver

--max-requests-inflight=2000
--max-mutating-requests-inflight=1000

2. kube-controller-manager

--kube-api-qps=100
--kube-api-burst=100

3. kube-scheduler

--kube-api-qps=200
--kube-api-burst=400

4. CloudCore

apiVersion: cloudcore.config.kubeedge.io/v1alpha1
kind: CloudCore
kubeAPIConfig:
  kubeConfig: ""
  master: ""
  qps: 60000
  burst: 80000
modules:
  cloudHub:
    advertiseAddress:
      - xx.xx.xx.xx
    nodeLimit: 30000
    tlsCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsCertFile: /etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    unixsocket:
      address: unix:///var/lib/kubeedge/kubeedge.sock
      enable: false
    websocket:
      address: 0.0.0.0
      enable: true
      port: 10000
  cloudStream:
    enable: false
  deviceController:
    enable: false
  dynamicController:
    enable: false
  edgeController:
    buffer:
      configMapEvent: 102400
      deletePod: 10240
      endpointsEvent: 1
      podEvent: 102400
      queryConfigMap: 10240
      queryEndpoints: 1
      queryNode: 10240
      queryPersistentVolume: 1
      queryPersistentVolumeClaim: 1
      querySecret: 10240
      queryService: 1
      queryVolumeAttachment: 1
      ruleEndpointsEvent: 1
      rulesEvent: 1
      secretEvent: 1
      serviceEvent: 10240
      updateNode: 15240
      updateNodeStatus: 30000
      updatePodStatus: 102400
    enable: true
    load:
      deletePodWorkers: 5000
      queryConfigMapWorkers: 1000
      queryEndpointsWorkers: 1
      queryNodeWorkers: 5000
      queryPersistentVolumeClaimWorkers: 1
      queryPersistentVolumeWorkers: 1
      querySecretWorkers: 1000
      queryServiceWorkers: 1
      queryVolumeAttachmentWorkers: 1
      updateNodeStatusWorkers: 10000
      updateNodeWorkers: 5000
      updatePodStatusWorkers: 20000
      ServiceAccountTokenWorkers: 10000
    nodeUpdateFrequency: 60
  router:
    enable: false
  syncController:
    enable: true

Density Test​

Test Execution​

Before using ClusterLoader2 to perform the performance test, we defined the test policy using the configuration file. In this test, we used the official Kubernetes density case. The configuration file we used can be obtained here:

https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/density/config.yaml

The following table describes the detailed Kubernetes resource configurations:

For details about the test method and procedure, see the following links:

https://github.com/kubeedge/kubeedge/tree/master/build/edgemark

https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md

Test Results​

APIResponsivenessPrometheusSimple：

1. mutating API latency（threshold=1s）：

   

2. Read-only API call latency(scope=resource, threshold=1s)

   

3. Read-only API call latency(scope=namespace, threshold=5s)

   

4. Read-only API call latency(scope=cluster, threshold=30s)

   

PodStartupLatency：

Note: Theoretically, the latency should always be greater than 0. Because kube-apiserver does not support RFC339NANO, the timestamp precision can only be seconds. Therefore, when the latency is low, some values collected by ClusterLoader2 are 0 due to precision loss.

Conclusion and Analysis​

From the preceding test results, the API call latency and pod startup latency meet the SLIs/SLOs defined by the Kubernetes community. Therefore, the KubeEdge-powered Kubernetes clusters can stably support 100,000 concurrent edge nodes and more than one million pods. In production, the network between edge nodes and the cloud is connected according to O&M requirements due to reasons such as network security and partition management. Therefore, the number of edge nodes that can be managed by a single cluster can increase proportionally based on the proportion of offline edge nodes to online ones. In addition, data fragmentation is used on the Kubernetes control plane to store different resources to the corresponding etcd space, which allows for a larger service deployment scale.

KubeEdge's Support for Large-Scale Edge Node Access​

1. Efficient Cloud-Edge Messaging​

List-watch is a unified mechanism for asynchronous messaging of Kubernetes components. The list operation calls the list API of a resource to obtain full resource data through non-persistent HTTP connections. The watch operation calls the watch API of a resource to monitor resource change events and obtain incremental change data through persistent HTTP connections and block-based transmission encoding. In Kubernetes, in addition to the list-watch of a node, pods allocated to the node, and full service metadata, kubelet must also watch (by default) the running pods mounted with secrets and ConfigMaps as data volumes. The number of list-watch operations could explode with increasing nodes and pods, which heavily burdens kube-apiserver.

KubeEdge uses the two-way multiplexing edge-cloud message channel and supports the WebSocket (default) and QUIC protocols. EdgeCore at the edge initiates a connection request to CloudCore on the cloud. CloudCore list-watches Kubernetes resource changes, and delivers metadata to the edge through this two-way channel. EdgeCore uploads the metadata, such as edge node status and application status, to CloudCore through this channel. CloudCore reports the received metadata to kube-apiserver.

CloudCore aggregates the upstream and downstream data. kube-apiserver processes only several list-watch requests from CloudCore. It can be effectively unburdened and the cluster performance gets improved.

Memory usage when the native Kubernetes kube-apiserver is used under the same node and pod scales:

Memory usage when kube-apiserver is used in a KubeEdge-powered Kubernetes cluster:

2. Reliable Incremental Cloud-Edge Data Transmission​

In the case of complex edge network topology or poor networking quality, cloud-edge communication may be compromised by high network latency, intermittent/frequent disconnection, and other issues. When the network recovers and edge nodes want to reconnect to the cloud, a large number of full list requests will be generated, pressuring kube-apiserver. Large-scale deployments may amplify this challenge to system stability. To solve it, KubeEdge records the version of the metadata successfully sent to the edge. When the cloud-edge network is reconnected, the cloud sends incremental metadata starting from the recorded metadata version.

3. Lightweight Edge + Edge-Cloud Messaging Optimization​

EdgeCore removes native kubelet features that are not used in edge deployments, such as in-tree volume and cloud-provider, trims the status information reported by nodes, and optimizes resource usage of edge agent software. EdgeCore can run with a minimum of 70 MB memory on edge devices whose memory is as minimum as 100 MB. The WebSocket channel, edge-cloud message combination, and data trim greatly reduces the communication pressure on the edge and cloud and the access pressure on the control plane. They ensure that the system can work properly even in the case of high latency and jitter.

When 100,000 edge nodes are connected, the number of ELB connections is 100,000.

When 100,000 edge nodes and more than 1,000,000 pods are deployed, the inbound rate of the ELB network is about 3 MB/s, and the average uplink bandwidth to each edge node is about 0.25 kbit/s.

Next Steps​

Targeted tests will be performed on edge devices, edge-cloud messaging, and edge service mesh. In addition, for some edge scenarios, such as large-scale node network disconnection and reconnection, high latency of edge networks, and intermittent disconnection, new SLIs/SLOs need to be introduced to measure the cluster service quality and perform large-scale tests.

The KubeEdge community attaches great importance to security and has set up Sig Security and Security Team to design KubeEdge system security and quickly respond to and handle security vulnerabilities. To conduct a more comprehensive security assessment of the KubeEdge project, the KubeEdge community cooperates with Ada Logics Ltd. and The Open Source Technology Improvement Fund performed a holistic security audit of KubeEdge and output a security auditing report, including the security threat model and security issues related to the KubeEdge project. Thank you to experts Adam Korczynski and David Korczynski of Ada Logics for their professional and comprehensive evaluation of the KubeEdge project, which has important guiding significance for the security protection of the KubeEdge project. Thank you Amir Montazery and Derek Zimmer of OSTIF and Cloud Native Computing Foundation (CNCF) who helped with this engagement.

The discovered security issues have been fixed and patched to the latest three minor release versions (v1.11.1, v1.10.2, v1.9.4) by KubeEdge maintainers according to the kubeedge security policy. Security advisories have been published here.

For more details of the threat model and the mitigations, Please check KubeEdge Threat Model And Security Protection Analysis: https://github.com/kubeedge/community/tree/master/sig-security/sig-security-audit/KubeEdge-threat-model-and-security-protection-analysis.md.

References:​

Audit report: https://github.com/kubeedge/community/tree/master/sig-security/sig-security-audit/KubeEdge-security-audit-2022.pdf

OSTIF Blogpost: https://ostif.org/our-audit-of-kubeedge-is-complete-multiple-security-issues-found-and-fixed

CNCF Blogpost:

KubeEdge Threat Model And Security Protection Analysis: https://github.com/kubeedge/community/tree/master/sig-security/sig-security-audit/KubeEdge-threat-model-and-security-protection-analysis.md

v1.11 What's New​

• Node Group Management
• Mapper SDK
• Beta sub-commands in Keadm to GA
• Deprecation of original init and join
• Next-gen Edged to Beta: Suitable for more scenarios

Release Highlights​

Node Group Management​

Users can now deploy applications to several node groups without writing deployment for every group. Node group management helps users to:

• Manage nodes in groups

• Spread apps among node groups

• Run different versions of app instances in different node groups

• Limit service endpoints in the same location as the client

Two new APIs have been introduced to implement Node Group Management:

• NodeGroup API: represents a group of nodes that have the same labels.
• EdgeApplication API: contains the template of the application organized by node groups, and the information on how to deploy different editions of the application to different node groups.

Refer to the links for more details (#3574, #3719).

Mapper SDK​

Mapper-sdk is a basic framework written in Go. Based on this framework, developers can more easily implement a new mapper. Mapper-sdk has realized the connection to KubeEdge, provides data conversion, and manages the basic properties and status of devices, etc., as well as basic capabilities and abstract definition of the driver interface. Developers only need to implement the customized protocol driver interface of the corresponding device to realize the function of mapper.

Refer to the link for more details (#70).

Beta sub-commands in Keadm to GA​

Some new sub-commands in Keadm have moved to GA, including containerized deployment, offline installation, etc. The original init and join behaviors have been replaced by the implementation from beta init and beta join:

• CloudCore will be running in containers and managed by Kubernetes Deployment by default.

• Keadm now downloads releases that are packed as container images to edge nodes for node setup.

• init: CloudCore Helm Chart is integrated into init, which can be used to deploy containerized CloudCore.

• join: Installing edgecore as a system service from a Docker image, no need to download from the GitHub release.

• reset: Reset the node, clean up the resources installed on the node by init or join. It will automatically detect the type of node to clean up.

• manifest generate: Generate all the manifests to deploy the cloud-side components.

Refer to the link for more details (#3900).

Deprecation of original init and join​

The original init and join sub-commands have been deprecated as they had issues with offline installation, etc.

Refer to the link for more details (#3900).

Next-gen Edged to Beta: Suitable for more scenarios​

The new version of the lightweight engine Edged, optimized from Kubelet and integrated into edgecore, has moved to Beta. The new Edged will still communicate with the cloud through the reliable transmission tunnel.

Refer to the link for more details (Dev-Branch for beta: feature-new-edged).

Important Steps before Upgrading​

If you want to use Keadm to deploy KubeEdge v1.11.0, please note that the behaviors of the init and join sub-commands have been changed.

Other Notable Changes​

• Add custom image repo for keadm join beta (#3654)

• Keadm: beta join support remote runtime (#3655)

• Use sync mode to update pod status (#3658)

• Make log level configurable for local up kubeedge (#3664)

• Use dependency to pull images (#3671)

• Move apis and client under kubeedge/cloud/pkg/ to kubeedge/pkg/ (#3683)

• Add subresource field in application for API with subresource (#3693)

• Add Keadm beta e2e (#3699)

• Keadm beta config images: support remote runtime (#3700)

• Use unified image management (#3720)

• Use armhf as default for armv7/v6 (#3723)

• Add ErrStatus in api-server application (#3742)

• Support compile binaries with kubeedge/build-tools image (#3756)

• Add min TLS version for stream server (#3764)

• Adding security policy (#3778)

• Chart: add cert domain config in helm chart (#3802)

• Add domain support for certgen.sh (#3808)

• Remove default KubeConfig for cloudcore (#3836)

• Helm: Allow annotation of the cloudcore service (#3856)

• Add rate limiter for edgehub (#3862)

• Sync pod status immediately when status update (#3891)

v1.10 What's New​

• Installation Experience Improvement with Keadm
• Preview version for Next-gen Edged: Suitable for more scenarios
• Edgemark: Support large-scale KubeEdge cluster performance testing
• EdgeMesh proxy tunnel supports quic
• EdgeMesh supports proxy for udp applications
• EdgeMesh support SSH login between cloud-edge/edge-edge nodes
• Kubernetes Dependencies Upgrade

Release Highlights​

Installation Experience Improvement with Keadm​

Keadm adds some new sub-commands to improve the user experience, including containerized deployment, offline installation, etc. New sub-commands including: beta, config.

beta provides some sub-commands that are still in testing, but have complete functions and can be used in advance. Sub-commands including: beta init, beta manifest generate, beta join, beta reset.

• beta init: CloudCore Helm Chart is integrated in beta init, which can be used to deploy containerized CloudCore.

• beta join: Installing edgecore as system service from docker image, no need to download from github release.

• beta reset: Reset the node, clean up the resources installed on the node by beta init or beta join. It will automatically detect the type of node to clean up.

• beta manifest generate: Generate all the manifests to deploy the cloudside components.

config is used to configure kubeedge cluster, like cluster upgrade, API conversion, image preloading. Now the image preloading has supported, sub-commands including: config images list, config images pull.

• config images list: List all images required for kubeedge installation.

• config images pull: Pull all images required for kubeedge installation.

Refer to the links for more details. (#3517, #3540, #3554, #3534)

Preview version for Next-gen Edged: Suitable for more scenarios​

A new version of the lightweight engine Edged, which is optimized from kubelet and integrated in edgecore, and occupies less resource. Users can customize lightweight optimization according to their needs.

Refer to the links for more details. (Dev-Branch for previewing: feature-new-edged)

Edgemark: Support large-scale KubeEdge cluster performance testing​

Edgemark is a performance testing tool inherited from Kubemark. The primary use case of Edgemark is also scalability testing, it allows users to simulate edge clusters, which can be much bigger than the real ones.

Edgemark consists of two parts: real cloud part components and a set of "Hollow" Edge Nodes. In "Hollow" Edge Nodes, EdgeCore runs in container. The edged module runs with an injected mock CRI part that doesn't do anything. So the hollow edge node doesn't actually start any containers, and also doesn't mount any volumes.

Refer to the link for more details. (#3637)

EdgeMesh proxy tunnel supports quic​

Users can choose edgemesh's proxy tunnel as quic protocol to transmit data. In edge scenarios, nodes are often in a weak network environment. Compared with the traditional tcp protocol, the quic protocol has better performance and QoS in the weak network environment.

Refer to the link for more details. (#281)

EdgeMesh supports proxy for udp applications​

Some users' services use the udp protocol, and now edgemesh can also support the proxy of udp applications.

Refer to the link for more details. (#295)

EdgeMesh support SSH login between cloud-edge/edge-edge nodes​

Edge nodes are generally distributed in the Private network environment, but it is often necessary to ssh login and operate the edge node. EdgeMesh provide a socks5proxy based on the tunnel inside EdgeMesh, which supports forwarding ssh requests from cloud/edge nodes to edge nodes.

Refer to the links for more details. (#258, #242)

Kubernetes Dependencies Upgrade​

Upgrade the vendered kubernetes version to v1.22.6, users now can use the feature of new version on the cloud and on the edge side.

Refer to the link for more details. (#3624)

Important Steps before Upgrading​

If you want to deploy the KubeEdge v1.10.0, please note that the Kubernetes dependency is 1.22.6.

Other Notable Changes​

• Remove dependency on os/exec and curl in favor of net/http (#3409, @mjlshen)

• Optimize script when create stream cert (#3412, @gujun4990)

• Cloudhub: prevent dropping volume messages (#3457, @moolen)

• Modify the log view command after edgecore is running (#3456, @zc2638)

• Optimize the iptables manager (#3461, @zhu733756)

• Add script for build release (#3467, @gy95)

• Using latest codes to do keadm_e2e (#3469, @gy95)

• Change the resourceType of msg issued by synccontroller (#3496, @Rachel-Shao)

• Add a basic image for building various components of KubeEdge (#3513, @zc2638)

• Supporting crossbuild all components (#3515, [@fisher

On December 6th, the KubeEdge community is proud to announce the availability of KubeEdge 1.9. This release includes a major upgrade for Custom HTTP Request Routing from Edge to Cloud through ServiceBus for Applications, CloudCore run independently of the Kubernetes Master host and containerized deployment using Helm, EdgeMesh add tls and encryption security, and compiled into rpm package, which includes:

• Custom HTTP Request Routing from Edge to Cloud through ServiceBus for Applications

• CloudCore run independently of the Kubernetes Master host

• EdgeMesh add tls and encryption security

• Enhance the ease of use of EdgeMesh

• Support containerized deployment of CloudCore using Helm

• Support compiled into rpm package and installed on OS such as openEuler using yum package manager

• 40+ bug fixes and enhancements.

Please refer to CHANGELOG v1.9 for a full list of features in this release

Release Highlights​

Support Custom HTTP Request Routing from Edge to Cloud through ServiceBus for Applications​

A HTTP server is added to ServiceBus, to support custom http request routing from edge to cloud for applications. This simplifies the rest api access with http server on the cloud while client is in the edge.

Refer to the links for more details. (#3254, #3301)

Support CloudCore to run independently of the Kubernetes Master host​

CloudCore now supports to run independently of the Kubernetes Master host, iptablesmanager has been added as an independent component, users only need to deploy the iptablesmanager to Kubernetes Master host, which now can add the iptable rules for Cloud-Edge tunnel automatically

Refer to the links for more details. (#3265)

EdgeMesh add tls and encryption security​

EdgeMesh's tunnel module adds tls and encryption security capabilities. These features bring more secure protection measures to the user's edgemesh-server component and reduce the risk of edgemesh-server being attacked.

Refer to the links for more details. (EdgeMesh#127)

Enhanced the ease of use of EdgeMesh​

EdgeMesh has many improvements in ease of use. Now users can easily deploy EdgeMesh's server and agent components with a single command of helm. At the same time, the restriction on service port naming is removed, and the docker0 dependency is removed, making it easier for users to use EdgeMesh.

Refer to the links for more details. (EdgeMesh#123, EdgeMesh#126, EdgeMesh#136, EdgeMesh#175)

Support containerized deployment of CloudCore using Helm​

CloudCore now supports containerized deployment using Helm, which provides better containerized deployment experience.

Refer to the links for more details. (#3265)

Support compiled into rpm package and installed on OS such as openEuler using yum package manager​

KubeEdge now supports compiled into rpm package and installed on OS such as openEuler using yum package manager.

Refer to the links for more details. (#3089, #3171)

In addition to the above new features, KubeEdge v1.9 also includes the following enhancements:

• Rpminstaller: add support for openEuler (#3089)

• Replaced 'kubeedge/pause' with multi arch image (#3114)

• Make meta server addr configurable (#3119)

• Added iptables to Dockerfile and made cloudcore privileged (#3129)

• Added CustomInterfaceEnabled and CustomInterfaceName for edgecore (#3130)

• Add experimental feature (#3131)

• Feat(edge): node ephemeral storage info (#3157)

• Support envFrom configmap in edge pods (#3176)

• Update golang to 1.16 (#3190)

• Metaserver: support shutdown server graceful (#3239)

• Support labelselector for metaserver (#3262)

Future Outlook​

With the release of v1.9, KubeEdge supports custom HTTP request routing from Edge to Cloud through ServiceBus for applications, supports CloudCore running independently of the Kubernetes Master host, supports containerized deployment of CloudCore using Helm, supports tls and encryption security and the ease of use of EdgeMesh. Thanks to Huawei, China Unicom, DaoCloud, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

On October 31st, the KubeEdge community is proud to announce the availability of KubeEdge 1.8. This release includes a major upgrade for Active-Active HA Support of CloudCore for Large Scale Cluster, EdgeMesh Architecture Modification, EdgeMesh Cross LAN Communication, and Kubernetes Dependencies Upgrade, which includes:

• Active-Active HA Support of CloudCore for Large Scale Cluster [Beta]

• EdgeMesh Architecture Modification

• EdgeMesh Cross LAN Communication

• Onvif Device Mapper

• Kubernetes Dependencies Upgrade

• 30+ bug fixes and enhancements.

Please refer to CHANGELOG v1.8 for a full list of features in this release

Release Highlights​

Active-Active HA Support of CloudCore for Large Scale Cluster [Beta]​

CloudCore now supports Active-Active HA mode deployment, which provides better scalability support for large scale clusters. Cloud-Edge tunnel can also work with multiple CloudCore instances. CloudCore now can add the iptable rules for Cloud-Edge tunnel automatically.

Refer to the links for more details. (#1560, #2999)

EdgeMesh Architecture Modification​

EdgeMesh now has two parts: edgemesh-server and edgemesh-agent. The edgemesh-server requires a public IP address, when users use cross lan communication, it can act as a relay server in the LibP2P mode or assist the agent to establish p2p hole punching. The edgemesh-agent is used to proxy all application traffic of user nodes, acts as an agent for communication between pods at different locations.

Refer to the links for more details. (edgemesh#19)

EdgeMesh Cross LAN Communication​

Users can use cross LAN communication feature to implement cross LAN edge to edge application communication and cross LAN edge to cloud application communication.

Refer to the links for more details. (edgemesh#26, edgemesh#37, edgemesh#57)

Onvif Device Mapper​

Onvif Device Mapper with Golang implementation is provided, based on new Device Mapper Standard. Users now can use onvif device mapper to manage the ONVIF IP camera.

Refer to the links for more details. (mappers-go#48)

Kubernetes Dependencies Upgrade​

Upgrade the vendered kubernetes version to v1.21.4, users now can use the feature of new version on the cloud and on the edge side.

Refer to the links for more details. (#3021, #3034)

In addition to the above new features, KubeEdge v1.8 also includes the following enhancements:

• Refactor edgesite: import functions and structs instead of copying code (#2893)

• Avoiding update cm after created a new cm (#2913)

• Solved the checksum file download problem when ke was installed offline (#2909)

• cloudcore support configmap dynamic update when the env of container inject from configmap or secret (#2931)

• Remove edgemesh from edgecore (#2916)

• keadm: support customsized labels when use join command (#2827)

• support k8s v1.21.X (#3021)

• Handling node/*/membership/detail (#3025)

• sync the response message unconditionally (#3014)

• support default NVIDIA SMI command (#2680)

Future Outlook​

With the release of v1.8, KubeEdge supports Active-Active HA mode deployment, which provides better scalability support for large scale clusters, supports cross LAN communication by EdgeMesh, and supports Onvif Device Mapper. Thanks to Huawei, China Unicom, DaoCloud, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

On June 10th, the KubeEdge community is proud to announce the availability of KubeEdge 1.7. This release includes a major upgrade for Active-Active HA Support of CloudCore for Large Cluster and a new Device Mapper Framework, which includes:

• Active-Active HA Support of CloudCore for Large Scale Cluster [Alpha]

• Support to manage Clusters on Edge [Alpha]

• Decoupled EdgeMesh from EdgeCore

• Autonomic Kube-API Endpoint for Applications On Edge Nodes [Beta]

• Custom HTTP Request Routing between Cloud and Edge for Applications [Alpha]

• Kubernetes Dependencies Upgrade

• 34+ bug fixes and enhancements.

Please refer to CHANGELOG v1.7 for a full list of features in this release

Release Highlights​

Active-Active HA Support of CloudCore for Large Scale Cluster [Alpha]​

CloudCore now supports Active-Active HA mode deployment, which provides better scalability support for large scale clusters. Cloud-Edge tunnel can also work with multiple CloudCore instances.

Refer to the links for more details. (#1560, #2867)

Support to manage Clusters on Edge [Alpha]​

In some scenarios, uses may have full-size Kubernetes clusters deployed on the edge. With EdgeSite, users are now able to access clusters on edge (in private network, behind NATed gateway, etc) from center cloud. (#2650, #2858)

Decoupled EdgeMesh from EdgeCore​

EdgeMesh aims to provide simplified network and services for edge applications. The EdgeMesh module is now decoupled from EdgeCore and able to be deployed as an independent components in containers.

Refer to EdgeMesh for more details

Mapper Framework​

Users are now able to use mapper framework to generate a new device mapper. This simplifies the mapper development when users trying to integrate with new protocols or new devices. (mappers-go#41)

Autonomic Kube-API Endpoint for Applications On Edge Nodes [Beta]​

Autonomic Kube-API Endpoint provides native Kubernetes API access on edge nodes. It's very useful in cases users want to run third-party plugins and applications that depends on Kubernetes APIs on edge nodes. With reliable message delivery and data autonomy provided by KubeEdge, list-watch connections on edge nodes keep available even when nodes are located in high latency network or frequently get disconnected to the Cloud.

In this release, a bunch of corner case issues are fixed and the stability is improved. And the feature maturity is now Beta.

Custom HTTP Request Routing between Cloud and Edge for Applications [Alpha]​

A new RuleEndpointType servicebus is added to RuleEndpoint API, to support custom http request routing between cloud and edge for applications. This simplifies the rest api access with http server on the edge while client is in the cloud. (#2588)

28+ bug fixes and enhancements​

In addition to the above new features, KubeEdge v1.7 also includes the following enhancements:

• Implement update rule status

• Install crd for router in keadm

• Remove synckeeper in edgehub

• upstream: refactor kubeClientGet

• make customsiz labels available when restart

Future Outlook​

With the release of v1.7, KubeEdge supports Active-Active HA mode deployment, which provides better scalability support for large scale clusters. Thanks to Huawei, China Unicom, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

KubeEdge v1.6: A major upgrade for maintainability​

On 27th February, the KubeEdge community is proud to announce the availability of KubeEdge 1.6. This release includes a major upgrade for maintainability, which includes:

• Support Autonomic Kube-API Endpoint for Applications On Edge Nodes [Alpha]

• Custom Message Routing between Cloud and Edge for Applications [Alpha]

• Simplified Application Autonomy Configuration When Node Is Off-line

• New home for Device Mappers code

• OPC-UA Device Mapper

• 24+ bug fixes and enhancements.

Please refer to CHANGELOG v1.6 for a full list of features in this release.

Release Highlights​

Support Autonomic Kube-API Endpoint for Applications On Edge Nodes [Alpha]​

Autonomic Kube-API Endpoint is now available on edge nodes! Users are now able to run third-party plugins and applications that depends on Kubernetes APIs on edge nodes. List-watch connections are established between client and the local endpoint provided by EdgeCore. With reliable message delivery and data autonomy provided by KubeEdge, list-watch connections on edge nodes keep available even when nodes are located in high latency network or frequently get disconnected to the Cloud.

This is very useful in cases that users want to install customized versions of Kubelet, Kube-Proxy, CNI and CSI plugins with KubeEdge. Particularly, Kubernetes CRDs are also supported on edge nodes. (#2508, #2587)

Custom Message Routing between Cloud and Edge for Applications [Alpha]​

Added support of routing management with Rule, RuleEndpoint API and a router module. Users are now able to use KubeEdge to deliver their custom messages between cloud and edge.

Note that it's designed for control data exchange between cloud and edge, not suitable for large data delivery. The data size of delivery at one time is limited to 12MB.

Refer to custom message deliver for more details. (#2413)

Simplified Application Autonomy Configuration When Node Is Off-line​

If user wants any application to stay on edge nodes when disconnected to the cloud, simply add label app-offline.kubeedge.io=autonomy to its pods. KubeEdge will automatically override pod default toleration configuration for Taint node.kubernetes.io/unreachable to avoid Kubernetes evicting pods from unreachable nodes. (#2499)

New home for Device Mappers code​

Device Mappers implementations now have a new home kubeedge/mappers-go.

OPC-UA Device Mapper​

OPC-UA Device Mapper with Golang implementation is provided, based on new Device Mapper Standard. (mappers-go#4).

24+ bug fixes and enhancements​

In addition to the above new features, KubeEdge v1.6 also includes the following enhancements:

• support kubectl get --raw /api/v1/nodes/{node}/proxy/metrics (#2437)

• support more metric path in cloud (#2482)

• edgecore: add nfs localpath support (#2529)

• add func that make subscribed topics persistence (#2457)

Future Outlook​

With the release of v1.6, KubeEdge provides autonomic Kube-API endpoint for applications On Edge nodes, custom message routing between Cloud and Edge, a new repository for Device Mappers, a more friendly user experience, and a more friendly community contributor experience. Thanks to Huawei, China Unicom, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

KubeEdge v1.5: A major upgrade for maintainability and edge devices management​

On 16th November, the KubeEdge community is proud to announce the availability of KubeEdge 1.5. This release includes a major upgrade for maintainability and edge devices management, which includes:

• Simplified Device Mapper reference architecture

• Modbus Mapper Golang Implementation

• Support Remote Exec to Pods on Edge From Cloud

• Support Keadm Debug Command for Trouble Shooting On Edge Nodes

• Kubernetes Dependencies Upgrade

• 23+ bug fixes and enhancements.

Please refer to https://github.com/kubeedge/kubeedge/blob/master/CHANGELOG/CHANGELOG-1.5.md for a full list of features in this release

Release Highlights​

Simplified Device Mapper reference architecture​

New version of Mapper reference architecture:

• Simplified Mapper code structure
• Extracted common code into SDK
• Added new building blocks: Configmap parser, Driver, Event process, Timer

Users are now able to develop mappers based on the new design standard.(#2147).

Modbus Mapper Golang Implementation​

A new modbus mapper with Golang implementation is provided, based on new Device Mapper Standard. (#2282).

Support Remote Exec to Pods on Edge From Cloud​

Users are now able to use K8s exec api or kubectl exec command to connect to pods on the edge node. (#2075).

Support Keadm Debug Command for Trouble Shooting On Edge Nodes​

A set of keadm debug subcommands are added for Trouble Shooting On Edge Nodes. Users are now able to use keadm debug get and keadm debug collect to get/collect KubeEdge local data for trouble shooting, and use keadm debug check and keadm debug diagnose to check local environment configuration. (#1939)

Kubernetes Dependencies Upgrade​

Upgrade the vendered kubernetes version to v1.19.3, users now can use the feature of new version on the cloud and on the edge side. (#2223)

Future Outlook​

With the release of v1.5, KubeEdge provides more complete device mapper, support for remote exec from Cloud to Pods on Edge, a more friendly user experience, and a more friendly community contributor experience. Thanks to Huawei, China Unicom, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

KubeEdge v1.4: A major upgrade for maintainability and edge devices management​

On August 15th, the KubeEdge community is proud to announce the availability of KubeEdge 1.4. This release includes a major upgrade for maintainability and edge devices management, which includes:

• Enhance Devices Management

• Metrics-Server Support for metrics collection across cloud and edge

• EdgeNode Certificate Rotation

• Kubernetes Dependencies Upgrade

• 34+ bug fixes and enhancements.

Please refer to https://github.com/kubeedge/kubeedge/blob/master/CHANGELOG/CHANGELOG-1.4.md for a full list of features in this release

Release Highlights​

Enhance Devices Management​

Upgrade device API from v1alpha1 to v1alpha2, enhancement include:

• A new field is added to explicitly provide customized protocol support
• Introduced data section to allow users defining data to get and process on the edge
• Moved propertyVistors from device model to Device instance API

Users are now able to customize the protocol of the edge device and also able to get and process the date in edge side.

Metrics-Server Support for metrics collection across cloud and edge​

With KubeEdge v1.4 users are now able to deploy metrics-server to collect resource metrics from edge nodes. Follow the instructions here to deploy the metrics-server.

Ref: https://github.com/kubeedge/kubeedge/blob/master/docs/setup/keadm.md#support-metrics-server-in-cloud

EdgeNode Certificate Rotation​

EdgeNodes are now able to automatically apply for new certificate when the certificate is about to expire and enforce TLS between CloudCore and EdgeCore.

Kubernetes Dependencies Upgrade​

Upgrade the venderod kubernetes version to v1.18.6, users now can use the feature of new version on the cloud and on the edge side.

34+ bug fixes and enhancements​

In addition to the above new features, KubeEdge v1.4 also includes the following enhancements:

• Add tree to store copy of dependency's license

• Add garbage collection of reliablesyncs when node unregisters

• Fix too long time to get node ready when reconnect

• Auto detect sandbox image

• Run edgecore as system service

Future Outlook​

With the release of v1.4, KubeEdge provides more complete edge application monitoring and management capabilities, a more stable and reliable cloud-side collaborative transmission mechanism, a more friendly user experience, and a more friendly community contributor experience. Thanks to Huawei, China Unicom, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

KubeEdge v1.3: A major upgrade for maintainability​

On May 15th, the KubeEdge community is proud to announce the availability of KubeEdge 1.3. This release includes a major upgrade for maintainability, which includes:

• Collecting logs from pods at edge in cloud

• Edge node and container monitoring

• High availability of KubeEdge cloud components

• Automated TLS bootstrapping for edge nodes

• CRI-O and Kata Containers runtime support

• 25+ bug fixes and enhancements.

Please refer to https://github.com/kubeedge/kubeedge/blob/master/CHANGELOG-1.3.md for a full list of features in this release

Release Highlights​

Collecting logs from pods at edge in cloud​

In most edge computing scenarios, the edge node is in a private network, and the pod logs running on the edge node cannot be directly pulled from the cloud, resulting issues for maintenance and debugging.

KubeEdge v1.3 includes a built-in streaming data channel which enables cloud to easily obtain edge application container logs via the kubectl logs command, without having to build another VPN server to solve private network access problems.

In addition, the KubeEdge community plans to provide a kubectl exec command support for edge containers in subsequent versions, so that users can easily connect to the edge application container from the cloud for debugging purposes.

See more feature details: https://docs.kubeedge.io/en/latest/setup/kubeedge_install_source.html

Edge node and container monitoring​

KubeEdge v1.3 provides a monitoring interface for edge nodes. Users can obtain edge node and its container information, and integrate it with third-party monitoring systems. This feature is enabled by default. Users have the option to disable this built-in monitoring module through the EnableMetrics item during configuration.

In the next version, KubeEdge will support the aggregation of edge node and application container monitoring information in the cloud.

See more feature details: https://github.com/kubeedge/kubeedge/pull/1573

High availability of KubeEdge cloud components​

In previous releases, the availability of KubeEdge cloud components rely on the automatic recovery mechanism of Kubernetes Deployment. In some extreme cases, this recovery can take a long time to recover from failures.

KubeEdge v1.3 has a built-in high-availability for the KubeEdge cloud component, CloudCore. When the CloudCore instance fails, a standby CloudCore instance is automatically switched on to minimize the impact of cloud component failures.

In subsequent versions, the KubeEdge community will further optimize the high concurrency of cloud components to improve throughput in large-scale edge nodes scenarios.

Automated TLS bootstrapping for edge nodes​

KubeEdge v1.3 introduced automated TLS bootstrapping for edge nodes, which simplifies the operation for users to configure cloud-edge secure channels and improves ease-of-use.

By default, KubeEdge generates a self-signed certificate for users, which is used for encrypted communication between cloud components and edge nodes. For scenarios that require an unified management of certificates, users can also use certificates issued by designated trust authorities.

For future releases, the KubeEdge community will support automatic renewal of the node's certificate after expiration.

See more feature details: https://github.com/kubeedge/kubeedge/blob/master/docs/setup/kubeedge_configure.md

More container runtime support​

KubeEdge v1.3 adds support of CRI-O and Kata Containers as container runtime.

• CRI-O, a CNCF incubation project, is a lightweight container, taking up to 30MB memory, and is in compliance with OCI standards.

• Kata Containers is an open source container runtime based on lightweight virtual machines. It is designed to combine the security advantages of virtual machines (VMs) with the speed and the manageability of containers.

With v1.3, KubeEdge has official support for all mainstream container runtimes including Docker, containerd, CRI-O and Kata Containers.

See more feature details: https://github.com/kubeedge/kubeedge/blob/master/docs/setup/kubeedge_cri_configure.md

25+ bug fixes and enhancements​

In addition to the above new features, KubeEdge v1.3 also includes the following enhancements:

• Added the support for keadm to install KubeEdge on CentOS systems

• EdgeMesh no longer depends on initContainer, and will take over traffic on the host during startup

• Fixed the issue that some pods in “the terminating state” cannot be deleted

Future Outlook​

With the release of v1.3, KubeEdge provides more complete edge application monitoring and management capabilities, a more stable and reliable cloud-side collaborative transmission mechanism, a more friendly user experience, and a more friendly community contributor experience. Thanks to Huawei, China Unicom, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use. Please refer to the roadmap document for future release plans:

https://github.com/kubeedge/kubeedge/blob/master/docs/getting-started/roadmap.md

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

Today we announce the v1.2 release of KubeEdge.​

On February 9th, the KubeEdge community is proud to announce the availability of KubeEdge 1.2. This release includes a major upgrade on reliability, which includes more reliable message delivery from cloud to edge, component Config API, edge nodes auto-registration, Kubernetes v1.17.1 support, and 30+ fixes.

Please refer to https://github.com/kubeedge/kubeedge/blob/release-1.2/CHANGELOG-1.2.md for a full list of features in this release, and the following for some highlights.

A major upgrade on Cloud-Edge transmission reliability​

In an Edge Computing scenario, the instability of edge network could cause the reliability issues of edge’s communication to cloud, which could further cause data loss during communication. To improve Cloud-Edge transmission reliability, KubeEdge v1.2 includes following update：

1. Added a verification mechanism for application layer message sending. There is a handshake mechanism built in to acknowledge (via ACK message) the successful state synchronization between cloud and edge. If the acknowledgement fails due to some reason, the application layer loop will trigger the retransmission mechanism to re-synchronize the state.
2. Implemented persistent cloud side collaborative messages. During the cloud-edge status synchronization process, cloud will record in real time the latest message version number (ResourceVersion) of each edge node that is successfully synchronized, and persist it to Kubernetes in the form of CRD. This mechanism can ensure the order and the continuity of message after a cloud failure or an edge node offline restart, avoiding cloud/edge inconsistent status caused by resending old messages.
3. Implemented a periodic check for cloud-edge data to ensure consistency. Based on the above two features, KubeEdge 1.2 includes a new module that synchronizes Controller to CloudCore. This module periodically checks the synchronization status of edge nodes, compares the information of resources in Kubernetes, and synchronizes states, and ensure the ultimate state consistency between cloud and edge.

See more details here: https://github.com/kubeedge/kubeedge/blob/master/docs/proposals/reliable-message-delivery.md

Component Config API​

An important improvement that KubeEdge v1.2 offers is the ability to update configuration of all components such as CloudCore, EdgeCore, EdgeSite, etc. by using the Kubernetes style Component Config API, and API versions to support backward compatibility.

In previous KubeEdge versions, the configuration of KubeEdge components were scattered in separate files for each module, so the maintenance was cumbersome. In this release, the team has aggregated all configurations, so users need to only update a configuration file, and the path of the configuration file can be easily set by using "- -config".

This release also offers two options for configuration: default configuration and minimum configuration. Users can use either option to generate configuration files and deploy KubeEdge quickly.

See more details here: https://github.com/kubeedge/kubeedge/pull/1172

Edge Nodes Auto-registration​

In previous versions, users needed to create a Node object on the cloud side first, and then start EdgeCore on the edge side when adding an edge node.

In release v1.2, KubeEdge provides the ability of automating edge node registration on the cloud side. This feature is enabled by default to improve ease of use. Users can turn it off by setting the "registerNode" configuration of EdgeCore to "false".

See more details here: https://github.com/kubeedge/kubeedge/pull/1401

Kubernetes v1.17.1 support​

KubeEdge v1.2 supports Kubernetes v1.17.1, so users can use the most recent Kubernetes application management, storage management etc.

Here is the full compatibility table: https://github.com/kubeedge/kubeedge

Other Fixes​

• Fixed CPU usage issues when EdgeCore runs multiple PODs
• Moved Beehive, Viaduct sub-projects under Staging directory for more friendly development experience
• More checks to insure EdgeCore and Kubelet are not running on the same host

Summary​

KubeEdge V1.2 offers a more stable and reliable cloud-edge transmission, enhanced edge application management and device management capabilities, a better user experience, and a more friendly community contributor experience.

A big “thank you” to all the community contributors and we hope to continue this momentum. Future releases of KubeEdge will further enhance KubeEdge user experience, better intelligent edge computing platform, and other advanced features to make KubeEdge a high performing, reliable, and intelligent solution to Connect Cloud to Edge.

For more details regarding KubeEdge, please follow and join us here: https://kubeedge.io .

Hearty Congratulations to all the winners. We will reach out to the winners soon via email.

That's right, contribute and win! Contribution is not limited to code contribution only; it can also include documentation, blogging, testing/issue identification, requirement identification and others. See details below

How to participate ?​

1. Raise pull request (PR) either for feature development / test code development (may be unit test code, edge module test code or end to end test code) in repos kubeedge / beehive / viaduct / website.
2. Identify defects, raise issues in respective repos kubeedge / beehive / viaduct / website.
3. Resolving existing issues in repos kubeedge / beehive / viaduct / website.
4. Share requirements by creating issues in repo kubeedge.
5. Writing blogs about KubeEdge either in the KubeEdge website (on PR approved & merged, this gets published in kubeedge.io website blog) or in other technical blogging site. Please refer here to know how to write a KubeEdge blog. Submit your blog details here.
6. Create your own sample applications and demo examples to illustrate possible use case(s) of using KubeEdge in repo examples.

Who can participate ?​

Anyone is welcome!

How the winners are selected ?​

Contribution can be made in the following various ways. Please see below for contribution requirements and how we select winners.

• Code contribution: Any code contribution should follow the contribution flow to get accepted. We will review the code submitted along with PR(s) for feature / test case development or issue fix.

• Issue identification: we will check the severity of issue and the quality of description that reproduces the identified issue with sufficient details.

• Requirement identification: we will check the quality of the requirement description, the uniqueness and the value of the identified requirement in comparison to the other Edge Computing platforms in the industry.

• KubeEdge project promotion: For any blog/wechat messages/twitter tweets/white papers/articles written about KubeEdge, we will review the content & popularity of the content.

• Example contribution: For any example created, we will review the code and the documentation of the steps & user guide.

Any contribution is greatly appreciated and 3 winners will be selected!

Timeline​

How the winners are notified ?​

Winners' Github ID will be published in this section. Winners will receive an e-mail that is associated with his/her Github ID. Any question, please contact us via:

• mailing list
• slack

Resources​

• kubeedge

• viaduct

• beehive

• examples

• website

KubeEdge community Code of Conduct​

KubeEdge follows the CNCF Code of conduct.

Edge computing framework capabilities should be able to cloud-native design patterns and practices such as container orchestration, microservices, serverless computation which has led to increasing heterogeneous deployment environments. Conventional practices for securing heterogeneous deployments add complexity overhead to enforcing policies, prevention and detection of threats. Due to the increase in complexity, there is more scope of error in manageability and also, constraints the scalability of the applications across multiple production environments. In such cases, a common identity framework for workloads becomes necessary to avoid the pit-falls of conventional security policies (such as managing network policies that are based on rules for traffic between particular ip addresses) which affect implementation of distributed patterns.

This enables to build a security model which is application-oriented rather than infrastructure-oriented.

What is SPIFFE and SPIRE?​

The SPIFFE standard provides a specification for a framework capable of bootstrapping and issuing identity to services across heterogeneous environments and organizational boundaries.

SPIFFE specification standardizes the process of assigning identities to workloads , verifying and validation of workload identities and workload API to retrieve the identities.

https://github.com/spiffe/spiffe

SPIFFE identities are encompassed in a SVID (SPIFFE Verifiable Identity Document). SVID specification provides the requirement for properties that must be supported when implementing SVID. Following link provides more information on SVID based on X509 certificate.

https://github.com/spiffe/spiffe/blob/master/standards/X509-SVID.md

SPIRE is a toolchain implementation for SPIFFE specification that enables establishing trust between workloads (using mTLS or JWT) across different deployment environments,issue SPIFFE IDs and workload API to retrieve workload SVIDs.

How does SPIRE work?​

Following information is extracted from Scytale presentations which gives informative and simplistic view on how SPIRE works.

What are the few desired security requirements for Kubeedge?​

Security is a paramount requirement for edge computing architecture as security breaches can make a complete organization to come to a halt (IIot) , data breach can lead to privacy issues and also control of the complete edge computing infrastructure. Few of the security requirements for deployment for kubeedge framework and edge application, but not limited to, are

• An identifiable edge node and workloads executing on the edge node.

• A method to verify the authenticity of the node and workloads executing on the node.

• Automated rotation of security credentials.

• Limit the affect of SPOF (in case of security-related events).

• Auditable security information about node and workloads in the environment.

• Limit access of user workloads to framework components and cloud.\

• Secure device provisioning.

• Device identity management and access control.

How SPIRE helps Kubeedge?​

• Node attestation: Only verifiable edge nodes can join the edge clusters. Every node is issued an identity on verification. In case of failed node attestations, no identity documents can be issued for services running on the node.

• Workload attestation: Only verifiable workload can run on edge nodes. In case of failed workload attestations, there are no identities issues for the workloads. All communications are blocked from unverified workloads.

• Certificate rotation: Short-lived certificates are generated and rotation policies can be configured for every service communication. There is no need for custom agents and reliance on specific orchestrators for certificate rotation configuration and management.

• Automated non-root CA certificate hierarchical deployments: Edge spire servers can be configured to not share any root CA chain for downstream nodes and workloads.