+
+This tutorial uses the federated learning dataset `FEMNIST` in the `leaf` dataset, which contains 62 different categories of handwritten digits and letters (digits 0 to 9, 26 lowercase letters, and 26 uppercase letters) with an image size of `28 x 28` pixels. The dataset contains handwritten digits and letters from 3500 users (up to 3500 clients can be simulated to participate in federated learning). The total data volume is 805,263, the average data volume per user is 226.83, and the variance of the data volume for all users is 88.94.
+
+## Device-Cloud Federated Learning Image Classification Dataset Process
+
+Refer to [leaf dataset instruction](https://github.com/TalwalkarLab/leaf) to download the dataset.
+
+1. Environmental requirements before downloading the dataset.
+
+ ```sh
+ numpy==1.16.4
+ scipy # conda install scipy
+ tensorflow==1.13.1 # pip install tensorflow
+ Pillow # pip install Pillow
+ matplotlib # pip install matplotlib
+ jupyter # conda install jupyter notebook==5.7.8 tornado==4.5.3
+ pandas # pip install pandas
+ ```
+
+2. Use git to download the official dataset generation script.
+
+ ```sh
+ git clone https://github.com/TalwalkarLab/leaf.git
+ ```
+
+ After downloading the project, the directory structure is as follows:
+
+ ```sh
+ leaf/data/femnist
+ ├── data # Used to store the dataset generated by the command
+ ├── preprocess # Store the code related to data pre-processing
+ ├── preprocess.sh # shell script generated by femnist dataset
+ └── README.md # Official dataset download guidance
+ ```
+
+3. Taking `femnist` dataset as an example, run the following command to enter the specified path.
+
+ ```sh
+ cd leaf/data/femnist
+ ```
+
+4. Using the command `. /preprocess.sh -s niid --sf 1.0 -k 0 -t sample` generates a dataset containing 3500 users, and the training sets and the test sets are divided in a ratio of 9:1 for each user's data.
+
+ The meaning of the parameters in the command can be found in the `leaf/data/femnist/README.md` file.
+
+ The directory structure after running is as follows:
+
+ ```text
+ leaf/data/femnist/35_client_sf1_data/
+ ├── all_data # All datasets are mixed together, without distinguishing the training sets and test sets, containing a total of 35 json files, and each json file contains the data of 100 users
+ ├── test # The test sets are divided into the training sets and the test sets in a ratio of 9:1 for each user's data, containing a total of 35 json files, and each json file contains the data of 100 users
+ ├── train # The training sets are divided into the training sets and the test sets in a ratio of 9:1 for each user's data, containing a total of 35 json files, and each json file contains the data of 100 users
+ └── ... # Other documents do not need to use, and details are not described herein
+ ```
+
+ Each json file contains the following three parts:
+
+ - `users`: User list.
+ - `num_samples`: The sample number list of each user.
+ - `user_data`: A dictionary object with user names as key and their respective data as value. For each user, the data is represented as a list of images, with each image represented as a list of integers of size 784 (obtained by spreading the `28 x 28` image array).
+
+ Before rerunning `preprocess.sh`, make sure to delete the `rem_user_data`, `sampled_data`, `test` and `train` subfolders from the data directory.
+
+5. Divide the 35 json files into 3500 json files (each json file represents a user).
+
+ The code is as follows:
+
+ ```python
+ import os
+ import json
+
+ def mkdir(path):
+ if not os.path.exists(path):
+ os.mkdir(path)
+
+ def partition_json(root_path, new_root_path):
+ """
+ partition 35 json files to 3500 json file
+
+ Each raw .json file is an object with 3 keys:
+ 1. 'users', a list of users
+ 2. 'num_samples', a list of the number of samples for each user
+ 3. 'user_data', an object with user names as keys and their respective data as values; for each user, data is represented as a list of images, with each image represented as a size-784 integer list (flattened from 28 by 28)
+
+ Each new .json file is an object with 3 keys:
+ 1. 'user_name', the name of user
+ 2. 'num_samples', the number of samples for the user
+ 3. 'user_data', an dict object with 'x' as keys and their respective data as values; with 'y' as keys and their respective label as values;
+
+ Args:
+ root_path (str): raw root path of 35 json files
+ new_root_path (str): new root path of 3500 json files
+ """
+ paths = os.listdir(root_path)
+ count = 0
+ file_num = 0
+ for i in paths:
+ file_num += 1
+ file_path = os.path.join(root_path, i)
+ print('======== process ' + str(file_num) + ' file: ' + str(file_path) + '======================')
+ with open(file_path, 'r') as load_f:
+ load_dict = json.load(load_f)
+ users = load_dict['users']
+ num_users = len(users)
+ num_samples = load_dict['num_samples']
+ for j in range(num_users):
+ count += 1
+ print('---processing user: ' + str(count) + '---')
+ cur_out = {'user_name': None, 'num_samples': None, 'user_data': {}}
+ cur_user_id = users[j]
+ cur_data_num = num_samples[j]
+ cur_user_path = os.path.join(new_root_path, cur_user_id + '.json')
+ cur_out['user_name'] = cur_user_id
+ cur_out['num_samples'] = cur_data_num
+ cur_out['user_data'].update(load_dict['user_data'][cur_user_id])
+ with open(cur_user_path, 'w') as f:
+ json.dump(cur_out, f)
+ f = os.listdir(new_root_path)
+ print(len(f), ' users have been processed!')
+ # partition train json files
+ partition_json("leaf/data/femnist/35_client_sf1_data/train", "leaf/data/femnist/3500_client_json/train")
+ # partition test json files
+ partition_json("leaf/data/femnist/35_client_sf1_data/test", "leaf/data/femnist/3500_client_json/test")
+ ```
+
+ where `root_path` is `leaf/data/femnist/35_client_sf1_data/{train,test}`. `new_root_path` is set by itself to store the generated 3500 user json files, which need to be processed separately for the training and test folders.
+
+ Each of the 3500 newly generated user json files contains the following three parts:
+
+ - `user_name`: User name.
+ - `num_samples`: The number of user samples
+ - `user_data`: A dictionary object with 'x' as key and user data as value; with 'y' as key and the label corresponding to the user data as value.
+
+ Print the result as following after running the script, which means a successful run:
+
+ ```sh
+ ======== process 1 file: /leaf/data/femnist/35_client_sf1_data/train/all_data_16_niid_0_keep_0_train_9.json======================
+ ---processing user: 1---
+ ---processing user: 2---
+ ---processing user: 3---
+ ......
+ ```
+
+6. Convert a json file to an image file.
+
+ Refer to the following code：
+
+ ```python
+ import os
+ import json
+ import numpy as np
+ from PIL import Image
+
+ name_list = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9',
+ 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U',
+ 'V', 'W', 'X', 'Y', 'Z',
+ 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u',
+ 'v', 'w', 'x', 'y', 'z'
+ ]
+
+ def mkdir(path):
+ if not os.path.exists(path):
+ os.mkdir(path)
+
+ def json_2_numpy(img_size, file_path):
+ """
+ read json file to numpy
+ Args:
+ img_size (list): contain three elements: the height, width, channel of image
+ file_path (str): root path of 3500 json files
+ return:
+ image_numpy (numpy)
+ label_numpy (numpy)
+ """
+ # open json file
+ with open(file_path, 'r') as load_f_train:
+ load_dict = json.load(load_f_train)
+ num_samples = load_dict['num_samples']
+ x = load_dict['user_data']['x']
+ y = load_dict['user_data']['y']
+ size = (num_samples, img_size[0], img_size[1], img_size[2])
+ image_numpy = np.array(x, dtype=np.float32).reshape(size) # mindspore doesn't support float64 and int64
+ label_numpy = np.array(y, dtype=np.int32)
+ return image_numpy, label_numpy
+
+ def json_2_img(json_path, save_path):
+ """
+ transform single json file to images
+
+ Args:
+ json_path (str): the path json file
+ save_path (str): the root path to save images
+
+ """
+ data, label = json_2_numpy([28, 28, 1], json_path)
+ for i in range(data.shape[0]):
+ img = data[i] * 255 # PIL don't support the 0/1 image ,need convert to 0~255 image
+ im = Image.fromarray(np.squeeze(img))
+ im = im.convert('L')
+ img_name = str(label[i]) + '_' + name_list[label[i]] + '_' + str(i) + '.png'
+ path1 = os.path.join(save_path, str(label[i]))
+ mkdir(path1)
+ img_path = os.path.join(path1, img_name)
+ im.save(img_path)
+ print('-----', i, '-----')
+
+ def all_json_2_img(root_path, save_root_path):
+ """
+ transform json files to images
+ Args:
+ json_path (str): the root path of 3500 json files
+ save_path (str): the root path to save images
+ """
+ usage = ['train', 'test']
+ for i in range(2):
+ x = usage[i]
+ files_path = os.path.join(root_path, x)
+ files = os.listdir(files_path)
+
+ for name in files:
+ user_name = name.split('.')[0]
+ json_path = os.path.join(files_path, name)
+ save_path1 = os.path.join(save_root_path, user_name)
+ mkdir(save_path1)
+ save_path = os.path.join(save_path1, x)
+ mkdir(save_path)
+ print('=============================' + name + '=======================')
+ json_2_img(json_path, save_path)
+
+ all_json_2_img("leaf/data/femnist/3500_client_json/", "leaf/data/femnist/3500_client_img/")
+ ```
+
+ Print the result as following after running the script, which means a successful run:
+
+ ```sh
+ =============================f0644_19.json=======================
+ ----- 0 -----
+ ----- 1 -----
+ ----- 2 -----
+ ......
+ ```
+
+7. Since the dataset under some user folders is small, if the number is smaller than the batch size, random expansion is required.
+
+ The entire dataset `"leaf/data/femnist/3500_client_img/"` can be checked and expanded by referring to the following code:
+
+ ```python
+ import os
+ import shutil
+ from random import choice
+
+ def count_dir(path):
+ num = 0
+ for root, dirs, files in os.walk(path):
+ for file in files:
+ num += 1
+ return num
+
+ def get_img_list(path):
+ img_path_list = []
+ label_list = os.listdir(path)
+ for i in range(len(label_list)):
+ label = label_list[i]
+ imgs_path = os.path.join(path, label)
+ imgs_name = os.listdir(imgs_path)
+ for j in range(len(imgs_name)):
+ img_name = imgs_name[j]
+ img_path = os.path.join(imgs_path, img_name)
+ img_path_list.append(img_path)
+ return img_path_list
+
+ def data_aug(data_root_path, batch_size = 32):
+ users = os.listdir(data_root_path)
+ tags = ["train", "test"]
+ aug_users = []
+ for i in range(len(users)):
+ user = users[i]
+ for tag in tags:
+ data_path = os.path.join(data_root_path, user, tag)
+ num_data = count_dir(data_path)
+ if num_data < batch_size:
+ aug_users.append(user + "_" + tag)
+ print("user: ", user, " ", tag, " data number: ", num_data, " < ", batch_size, " should be aug")
+ aug_num = batch_size - num_data
+ img_path_list = get_img_list(data_path)
+ for j in range(aug_num):
+ img_path = choice(img_path_list)
+ info = img_path.split(".")
+ aug_img_path = info[0] + "_aug_" + str(j) + ".png"
+ shutil.copy(img_path, aug_img_path)
+ print("[aug", j, "]", "============= copy file:", img_path, "to ->", aug_img_path)
+ print("the number of all aug users: " + str(len(aug_users)))
+ print("aug user name: ", end=" ")
+ for k in range(len(aug_users)):
+ print(aug_users[k], end = " ")
+
+ if __name__ == "__main__":
+ data_root_path = "leaf/data/femnist/3500_client_img/"
+ batch_size = 32
+ data_aug(data_root_path, batch_size)
+ ```
+
+8. Convert the expanded image dataset into a bin file format usable in the Federated Learning Framework.
+
+ Refer to the following code:
+
+ ```python
+ import numpy as np
+ import os
+ import mindspore.dataset as ds
+ import mindspore.dataset.vision as vision
+ import mindspore.dataset.transforms as transforms
+ import mindspore
+
+ def mkdir(path):
+ if not os.path.exists(path):
+ os.mkdir(path)
+
+ def count_id(path):
+ files = os.listdir(path)
+ ids = {}
+ for i in files:
+ ids[i] = int(i)
+ return ids
+
+ def create_dataset_from_folder(data_path, img_size, batch_size=32, repeat_size=1, num_parallel_workers=1, shuffle=False):
+ """ create dataset for train or test
+ Args:
+ data_path: Data path
+ batch_size: The number of data records in each group
+ repeat_size: The number of replicated data records
+ num_parallel_workers: The number of parallel workers
+ """
+ # define dataset
+ ids = count_id(data_path)
+ mnist_ds = ds.ImageFolderDataset(dataset_dir=data_path, decode=False, class_indexing=ids)
+ # define operation parameters
+ resize_height, resize_width = img_size[0], img_size[1] # 32
+
+ transform = [
+ vision.Decode(True),
+ vision.Grayscale(1),
+ vision.Resize(size=(resize_height, resize_width)),
+ vision.Grayscale(3),
+ vision.ToTensor(),
+ ]
+ compose = transforms.Compose(transform)
+
+ # apply map operations on images
+ mnist_ds = mnist_ds.map(input_columns="label", operations=transforms.TypeCast(mindspore.int32))
+ mnist_ds = mnist_ds.map(input_columns="image", operations=compose)
+
+ # apply DatasetOps
+ buffer_size = 10000
+ if shuffle:
+ mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size) # 10000 as in LeNet train script
+ mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)
+ mnist_ds = mnist_ds.repeat(repeat_size)
+ return mnist_ds
+
+ def img2bin(root_path, root_save):
+ """
+ transform images to bin files
+
+ Args:
+ root_path: the root path of 3500 images files
+ root_save: the root path to save bin files
+
+ """
+
+ use_list = []
+ train_batch_num = []
+ test_batch_num = []
+ mkdir(root_save)
+ users = os.listdir(root_path)
+ for user in users:
+ use_list.append(user)
+ user_path = os.path.join(root_path, user)
+ train_test = os.listdir(user_path)
+ for tag in train_test:
+ data_path = os.path.join(user_path, tag)
+ dataset = create_dataset_from_folder(data_path, (32, 32, 1), 32)
+ batch_num = 0
+ img_list = []
+ label_list = []
+ for data in dataset.create_dict_iterator():
+ batch_x_tensor = data['image']
+ batch_y_tensor = data['label']
+ trans_img = np.transpose(batch_x_tensor.asnumpy(), [0, 2, 3, 1])
+ img_list.append(trans_img)
+ label_list.append(batch_y_tensor.asnumpy())
+ batch_num += 1
+
+ if tag == "train":
+ train_batch_num.append(batch_num)
+ elif tag == "test":
+ test_batch_num.append(batch_num)
+
+ imgs = np.array(img_list) # (batch_num, 32,3,32,32)
+ labels = np.array(label_list)
+ path1 = os.path.join(root_save, user)
+ mkdir(path1)
+ image_path = os.path.join(path1, user + "_" + "bn_" + str(batch_num) + "_" + tag + "_data.bin")
+ label_path = os.path.join(path1, user + "_" + "bn_" + str(batch_num) + "_" + tag + "_label.bin")
+
+ imgs.tofile(image_path)
+ labels.tofile(label_path)
+ print("user: " + user + " " + tag + "_batch_num: " + str(batch_num))
+ print("total " + str(len(use_list)) + " users finished!")
+
+ root_path = "leaf/data/femnist/3500_client_img/"
+ root_save = "leaf/data/femnist/3500_clients_bin"
+ img2bin(root_path, root_save)
+ ```
+
+ Print the result as following after running the script, which means a successful run:
+
+ ```sh
+ user: f0141_43 test_batch_num: 1
+ user: f0141_43 train_batch_num: 10
+ user: f0137_14 test_batch_num: 1
+ user: f0137_14 train_batch_num: 11
+ ......
+ total 3500 users finished!
+ ```
+
+9. Generate `3500_clients_bin` folder containing a total of 3500 user folders with the following directory structure:
+
+ ```sh
+ leaf/data/femnist/3500_clients_bin
+ ├── f0000_14 # User number
+ │ ├── f0000_14_bn_10_train_data.bin # The training data of user f0000_14 (The number 10 after bn_ represents the batch number)
+ │ ├── f0000_14_bn_10_train_label.bin # Training tag for user f0000_14
+ │ ├── f0000_14_bn_1_test_data.bin # Test data of user f0000_14 (the number 1 after bn_ represents batch number)
+ │ └── f0000_14_bn_1_test_label.bin # Test tag for user f0000_14
+ ├── f0001_41 # User number
+ │ ├── f0001_41_bn_11_train_data.bin # The training data of user f0001_41 (The number 11 after bn_ represents the batch number)
+ │ ├── f0001_41_bn_11_train_label.bin # Training tag for user f0001_41
+ │ ├── f0001_41_bn_1_test_data.bin # Test data of user f0001_41 (the number 1 after bn_ represents batch number)
+ │ └── f0001_41_bn_1_test_label.bin # Test tag for user f0001_41
+ │ ...
+ └── f4099_10 # User number
+ ├── f4099_10_bn_4_train_data.bin # The training data of user f4099_10 (the number 4 after bn_ represents the batch number)
+ ├── f4099_10_bn_4_train_label.bin # Training tag for user f4099_10
+ ├── f4099_10_bn_1_test_data.bin # Test data of user f4099_10 (the number 1 after bn_ represents batch number)
+ └── f4099_10_bn_1_test_label.bin # Test tag for user f4099_10
+ ```
+
+The `3500_clients_bin` folder generated according to steps 1 to 9 above can be directly used as the input data for the device-cloud federated image classification task.
diff --git a/docs/federated/docs/source_en/index.rst b/docs/federated/docs/source_en/index.rst
index 39f7e69a8f..9c76cd68c9 100644
--- a/docs/federated/docs/source_en/index.rst
+++ b/docs/federated/docs/source_en/index.rst
@@ -108,4 +108,5 @@ Common Application Scenarios
:maxdepth: 1
:caption: References
+ image_classfication_dataset_process
faq
\ No newline at end of file
diff --git a/docs/federated/docs/source_zh_cn/image_classfication_dataset_process.md b/docs/federated/docs/source_zh_cn/image_classfication_dataset_process.md
index e7fec758a0..18ff515ef6 100644
--- a/docs/federated/docs/source_zh_cn/image_classfication_dataset_process.md
+++ b/docs/federated/docs/source_zh_cn/image_classfication_dataset_process.md
@@ -2,7 +2,7 @@
-本教程采用`leaf`数据集中的联邦学习数据集`FEMNIST`， 该数据集包含62个不同类别的手写数字和字母（数字0~9、26个小写字母、26个大写字母），图像大小为`28 x 28`像素，数据集包含3500个用户的手写数字和字母（最多可模拟3500个客户端参与联邦学习），总数据量为805263，平均每个用户包含数据量为226.83，所有用户数据量的方差为88.94。
+本教程采用`leaf`数据集中的联邦学习数据集`FEMNIST`，该数据集包含62个不同类别的手写数字和字母（数字0~9、26个小写字母、26个大写字母），图像大小为`28 x 28`像素，数据集包含3500个用户的手写数字和字母（最多可模拟3500个客户端参与联邦学习），总数据量为805263，平均每个用户包含数据量为226.83，所有用户数据量的方差为88.94。
## 端云联邦学习图像分类数据集处理
@@ -132,7 +132,7 @@
- `user_name`: 用户名。
- `num_samples`: 用户的样本数。
- - `user_data`: 一个以'x'为key，以用户数据为value的字典对象； 以'y'为key，以用户数据对应的标签为value。
+ - `user_data`: 一个以'x'为key，以用户数据为value的字典对象；以'y'为key，以用户数据对应的标签为value。
运行该脚本打印如下，代表运行成功：
@@ -434,18 +434,18 @@
├── f0000_14 # 用户编号
│ ├── f0000_14_bn_10_train_data.bin # 用户f0000_14的训练数据 （bn_后面的数字10代表batch number）
│ ├── f0000_14_bn_10_train_label.bin # 用户f0000_14的训练标签
- │ ├── f0000_14_bn_1_test_data.bin # 用户f0000_14的测试数据 （bn_后面的数字1代表batch number）
+ │ ├── f0000_14_bn_1_test_data.bin # 用户f0000_14的测试数据 （bn_后面的数字1代表batch number）
│ └── f0000_14_bn_1_test_label.bin # 用户f0000_14的测试标签
├── f0001_41 # 用户编号
│ ├── f0001_41_bn_11_train_data.bin # 用户f0001_41的训练数据 （bn_后面的数字11代表batch number）
│ ├── f0001_41_bn_11_train_label.bin # 用户f0001_41的训练标签
- │ ├── f0001_41_bn_1_test_data.bin # 用户f0001_41的测试数据 （bn_后面的数字1代表batch number）
- │ └── f0001_41_bn_1_test_label.bin # 用户f0001_41的测试标签
+ │ ├── f0001_41_bn_1_test_data.bin # 用户f0001_41的测试数据 （bn_后面的数字1代表batch number）
+ │ └── f0001_41_bn_1_test_label.bin # 用户f0001_41的测试标签
│ ...
└── f4099_10 # 用户编号
├── f4099_10_bn_4_train_data.bin # 用户f4099_10的训练数据 （bn_后面的数字4代表batch number）
├── f4099_10_bn_4_train_label.bin # 用户f4099_10的训练标签
- ├── f4099_10_bn_1_test_data.bin # 用户f4099_10的测试数据 （bn_后面的数字1代表batch number）
+ ├── f4099_10_bn_1_test_data.bin # 用户f4099_10的测试数据 （bn_后面的数字1代表batch number）
└── f4099_10_bn_1_test_label.bin # 用户f4099_10的测试标签
```
diff --git a/docs/federated/docs/source_zh_cn/image_classification_application_in_cross_silo.md b/docs/federated/docs/source_zh_cn/image_classification_application_in_cross_silo.md
index 2fec556f21..a332654cfd 100644
--- a/docs/federated/docs/source_zh_cn/image_classification_application_in_cross_silo.md
+++ b/docs/federated/docs/source_zh_cn/image_classification_application_in_cross_silo.md
@@ -8,7 +8,7 @@
## 下载数据集
-本示例采用[leaf数据集](https://github.com/TalwalkarLab/leaf)中的联邦学习数据集`FEMNIST`， 该数据集包含62个不同类别的手写数字和字母（数字0~9、26个小写字母、26个大写字母），图像大小为`28 x 28`像素，数据集包含3500个用户的手写数字和字母（最多可模拟3500个客户端参与联邦学习），总数据量为805263，平均每个用户包含数据量为226.83，所有用户数据量的方差为88.94。
+本示例采用[leaf数据集](https://github.com/TalwalkarLab/leaf)中的联邦学习数据集`FEMNIST`，该数据集包含62个不同类别的手写数字和字母（数字0~9、26个小写字母、26个大写字母），图像大小为`28 x 28`像素，数据集包含3500个用户的手写数字和字母（最多可模拟3500个客户端参与联邦学习），总数据量为805263，平均每个用户包含数据量为226.83，所有用户数据量的方差为88.94。
可参考文档[端云联邦学习图像分类数据集处理](https://www.mindspore.cn/federated/docs/zh-CN/master/image_classfication_dataset_process.html)中步骤1~7获取图片形式的3500个用户数据集`3500_client_img`。
@@ -153,7 +153,7 @@ if __name__ == "__main__":
### 安装MindSpore和Mindspore Federated
-包括源码和下载发布版两种方式，支持CPU、GPU硬件平台，根据硬件平台选择安装即可。安装步骤可参考[MindSpore安装指南](https://www.mindspore.cn/install)， [Mindspore Federated安装指南](https://www.mindspore.cn/federated/docs/zh-CN/master/federated_install.html)。
+包括源码和下载发布版两种方式，支持CPU、GPU硬件平台，根据硬件平台选择安装即可。安装步骤可参考[MindSpore安装指南](https://www.mindspore.cn/install)，[Mindspore Federated安装指南](https://www.mindspore.cn/federated/docs/zh-CN/master/federated_install.html)。
目前联邦学习框架只支持Linux环境中部署，cross-silo联邦学习框架需要MindSpore版本号>=1.5.0。
diff --git a/docs/mindspore/source_en/index.rst b/docs/mindspore/source_en/index.rst
index 8cf01bc7b6..9163308ef1 100644
--- a/docs/mindspore/source_en/index.rst
+++ b/docs/mindspore/source_en/index.rst
@@ -76,7 +76,10 @@ MindSpore Documentation
migration_guide/overview
migration_guide/enveriment_preparation
+ migration_guide/analysis_and_preparation
migration_guide/model_development/model_development
+ migration_guide/debug_and_tune
+ migration_guide/sample_code
migration_guide/use_third_party_op
.. toctree::
diff --git a/docs/mindspore/source_en/migration_guide/analysis_and_preparation.md b/docs/mindspore/source_en/migration_guide/analysis_and_preparation.md
new file mode 100644
index 0000000000..2925e33cce
--- /dev/null
+++ b/docs/mindspore/source_en/migration_guide/analysis_and_preparation.md
@@ -0,0 +1,475 @@
+# Model Analysis and Preparation
+
+
+
+## Obtaining Sample Code
+
+When you obtain a paper to implement migration on MindSpore, you need to find the reference code that has been implemented in other frameworks. In principle, the reference code must meet at least one of the following requirements:
+
+1. The author opens the paper to the public.
+2. The implementation is starred and forked by many developers, which means it is widely recognized.
+3. The code is new and maintained by developers.
+4. The PyTorch reference code is preferred.
+
+If a new paper has no reference implementation, you can refer to [Constructing MindSpore Network](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/model_development.html).
+
+## Analyzing Algorithm and Network Structure
+
+First, when reading the paper and reference code, you need to analyze the network structure to organize the code writing. The following shows the general network structure of YOLOX.
+
+| Module| Implementation|
+| ---- | ---- |
+| backbone | CSPDarknet (s, m, l, x)|
+| neck | FPN |
+| head | Decoupled Head |
+
+Second, analyze the innovative points of the migration algorithm and record the tricks used during the training, for example, data augmentation added during data processing, shuffle, optimizer, learning rate attenuation policy, and parameter initialization. You can prepare a checklist and fill in the corresponding items during analysis.
+
+For example, the following records some tricks used by the YOLOX network during training.
+
+| Trick | +Record | +
|---|---|
| Data augmentation | +Mosaic, including random scaling, crop, and layout | +
| MixUp | +|
| Learning rate attenuation policy | +Multiple attenuation modes are available. By default, the COS learning rate attenuation is used. | +
| Optimizer parameters | +SGD momentum=0.9, nesterov=True, and no weight decay | +
| Training parameters | +epoch: 300; batchsize: 8 | +
| Network structure optimization points | +Decoupled Head; Anchor Free; SimOTA | +
| Training process optimization points | +EMA; Data augmentation is not performed for the last 15 epochs; mixed precision | +
+
+## Function Debugging
+
+During network migration, you are advised to use the PyNative mode for debugging. In PyNative mode, you can perform debugging, and log printing is user-friendly. After the debugging is complete, the graph mode is used. The graph mode is more user-friendly in execution performance. You can also find some problems in network compilation. For example, gradient truncation caused by third-party operators.
+For details, see [Function Debugging](https://www.mindspore.cn/tutorials/experts/en/master/debug/function_debug.html).
+
+## Accuracy Debugging
+
+The accuracy debugging process is as follows:
+
+### 1. Checking Parameters
+
+This part includes checking all parameters and the number of trainable parameters, and checking the shape of all parameters.
+
+#### Obtaining MindSpore Parameters
+
+`Parameter` is used for MindSpore trainable and untrainable parameters.
+
+```python
+from mindspore import nn
+
+class msNet(nn.Cell):
+ def __init__(self):
+ super(msNet, self).__init__()
+ self.fc = nn.Dense(1, 1, weight_init='normal')
+ def construct(self, x):
+ output = self.fc(x)
+ return output
+
+msnet = msNet()
+# Obtain all parameters.
+all_parameter = []
+for item in msnet.get_parameters():
+ all_parameter.append(item)
+ print(item.name, item.data.shape)
+print(f"all parameter numbers: {len(all_parameter)}")
+
+# Obtain trainable parameters.
+trainable_params = msnet.trainable_params()
+for item in trainable_params:
+ print(item.name, item.data.shape)
+print(f"trainable parameter numbers: {len(trainable_params)}")
+```
+
+```text
+ fc.weight (1, 1)
+ fc.bias (1,)
+ all parameter numbers: 2
+ fc.weight (1, 1)
+ fc.bias (1,)
+ trainable parameter numbers: 2
+```
+
+#### Obtaining PyTorch Parameters
+
+`Parameter` is used for PyTorch trainable parameters, and `requires_grad=False` or `buffer` is used for PyTorch untrainable parameters.
+
+```python
+from torch import nn
+
+class ptNet(nn.Module):
+ def __init__(self):
+ super(ptNet, self).__init__()
+ self.fc = nn.Linear(1, 1)
+ def construct(self, x):
+ output = self.fc(x)
+ return output
+
+
+ptnet = ptNet()
+all_parameter = []
+trainable_params = []
+# Obtain network parameters.
+for name, item in ptnet.named_parameters():
+ if item.requires_grad:
+ trainable_params.append(item)
+ all_parameter.append(item)
+ print(name, item.shape)
+
+for name, buffer in ptnet.named_buffers():
+ all_parameter.append(buffer)
+ print(name, buffer.shape)
+print(f"all parameter numbers: {len(all_parameter)}")
+print(f"trainable parameter numbers: {len(trainable_params)}")
+```
+
+```text
+ fc.weight torch.Size([1, 1])
+ fc.bias torch.Size([1])
+ all parameter numbers: 2
+ trainable parameter numbers: 2
+```
+
+The parameters of MindSpore and PyTorch are similar except BatchNorm. Note that MindSpore does not have parameters corresponding to `num_batches_tracked`. You can replace this parameter with `global_step` in the optimizer.
+
+| MindSpore | PyTorch |
+| --------- | --------|
+| gamma | weight |
+| beta | bias |
+| moving_mean | running_mean |
+| moving_variance | running_var |
+| -| num_batches_tracked |
+
+### 2. Model Verification
+
+The implementation of the model algorithm is irrelevant to the framework. The trained parameters can be converted into the [checkpoint](https://www.mindspore.cn/tutorials/en/master/beginner/save_load.html) file of MindSpore and loaded to the network for inference verification.
+
+For details about the model verification process, see [ResNet Network Migration](https://www.mindspore.cn/docs/en/master/migration_guide/sample_code.html#model-validation).
+
+### 3. Inference Verification
+
+After confirming that the model structures are the same, you are advised to perform inference verification again. In addition to models, the entire inference process also involves datasets and metrics. When the inference results are inconsistent, you can use the control variable method to gradually rectify the fault.
+
+For details about the inference verification process, see [ResNet Network Migration](https://www.mindspore.cn/docs/en/master/migration_guide/sample_code.html#inference-process).
+
+### 4. Training Accuracy
+
+After the inference verification is complete, the basic model, data processing, and metrics calculation are normal. If the training accuracy is still abnormal, how do we locate the fault?
+
+- Add loss scale. On Ascend, operators such as Conv, Sort, and TopK can only be float16. MatMul is recommended to be float16 due to performance problems. Therefore, it is recommended that loss scale be used as a standard configuration for network training.
+
+```python
+import mindspore as ms
+from mindspore import nn
+# Model
+loss_scale_manager = ms.FixedLossScaleManager(drop_overflow_update=False) # Static loss scale
+# loss_scale_manager = ms.DynamicLossScaleManager() # Dynamic loss scale
+
+# 1. General process
+loss = nn.MSELoss()
+opt = nn.Adam(params=msnet.trainable_params(), learning_rate=0.01)
+model = ms.Model(network=msnet, loss_fn=loss, optimizer=opt, loss_scale_manager=loss_scale_manager)
+
+# 2. Self-packaged forward network and loss function
+msnet.to_float(ms.float16)
+loss.to_float(ms.float32)
+net_with_loss = nn.WithLossCell(msnet, loss)
+# It is recommended that loss_fn be used for the mixed precision of the model. Otherwise, float16 is used for calculation of the loss part, which may cause overflow.
+model = ms.Model(network=net_with_loss, optimizer=opt)
+
+# 3. Self-packaged training process
+scale_sense = nn.FixedLossScaleUpdateCell(1)#(config.loss_scale) # Static loss scale
+# scale_sense = nn.DynamicLossScaleUpdateCell(loss_scale_value=config.loss_scale,
+# scale_factor=2, scale_window=1000) # Dynamic loss scale
+train_net = nn.TrainOneStepWithLossScaleCell(net_with_loss, optimizer=opt, scale_sense=scale_sense)
+model = ms.Model(network=train_net)
+```
+
+- Check whether overflow occurs. When loss scale is added, overflow detection is added by default to monitor the overflow result. If overflow occurs continuously, you are advised to use the [debugger](https://www.mindspore.cn/mindinsight/docs/en/master/debugger.html) or [dump data](https://mindspore.cn/tutorials/experts/en/master/debug/dump.html) of MindInsight to check why overflow occurs.
+
+```python
+import numpy as np
+from mindspore import dataset as ds
+
+def get_data(num, w=2.0, b=3.0):
+ for _ in range(num):
+ x = np.random.uniform(-10.0, 10.0)
+ noise = np.random.normal(0, 1)
+ y = x * w + b + noise
+ yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)
+
+
+def create_dataset(num_data, batch_size=16, repeat_size=1):
+ input_data = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])
+ input_data = input_data.batch(batch_size, drop_remainder=True)
+ input_data = input_data.repeat(repeat_size)
+ return input_data
+
+train_net.set_train()
+dataset = create_dataset(1600)
+iterator = dataset.create_tuple_iterator()
+for i, data in enumerate(iterator):
+ loss, overflow, scaling_sens = train_net(*data)
+ print("step: {}, loss: {}, overflow:{}, scale:{}".format(i, loss, overflow, scaling_sens))
+```
+
+```text
+ step: 0, loss: 138.42825, overflow:False, scale:1.0
+ step: 1, loss: 118.172104, overflow:False, scale:1.0
+ step: 2, loss: 159.14542, overflow:False, scale:1.0
+ step: 3, loss: 150.65671, overflow:False, scale:1.0
+ ... ...
+ step: 97, loss: 69.513245, overflow:False, scale:1.0
+ step: 98, loss: 51.903114, overflow:False, scale:1.0
+ step: 99, loss: 42.250656, overflow:False, scale:1.0
+```
+
+- Check the optimizer, loss, and parameter initialization. In addition to the model and dataset, only the optimizer, loss, and parameter initialization are added in the entire training process. If the training is abnormal, check the optimizer, loss, and parameter initialization. Especially for loss and parameter initialization, there is a high probability that the problem occurs.
+- Check whether to add seeds for multiple devices to ensure that the initialization of multiple SIM cards is consistent. Determine whether to perform gradient aggregation during [customized training](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/training_and_gradient.html#customizing-training-cell).
+
+```python
+import mindspore as ms
+ms.set_seed(1) # The random seeds of MindSpore, NumPy, and dataset are fixed. The random seed of the API needs to be set in the API attribute.
+```
+
+- Check whether the data processing meets the expectation through visualization. Focus on data shuffle and check whether data mismatch occurs.
+
+For details about more accuracy debugging policies, see [Accuracy Debugging](https://mindspore.cn/mindinsight/docs/en/master/accuracy_problem_preliminary_location.html).
+
+## Performance Tuning
+
+The performance tuning directions are as follows:
+
+1. Operator performance tuning
+2. Framework enabling performance tuning
+3. Multi-Node synchronization performance tuning
+4. Data processing performance tuning
+
+For details, see [ResNet Network Migration](https://www.mindspore.cn/docs/en/master/migration_guide/sample_code.html).
+
+> Some networks are large or there are many [process control statements](https://mindspore.cn/tutorials/en/master/advanced/modules/control_flow.html). In this case, the build is slow in graph mode. During performance tuning, distinguish graph build from network execution. This section describes the performance tuning policies in the network execution phase. If graph build is slow, try [incremental operator build](https://mindspore.cn/tutorials/experts/en/master/debug/op_compilation.html) or contact [MindSpore community](https://gitee.com/mindspore/mindspore/issues) for feedback.
+
+### Operator Performance Tuning
+
+#### Poor Operator Performance
+
+If a single operator takes a long time and the performance of the same operator varies greatly in different shapes or data types, the problem is caused by the operator performance. The solution is as follows:
+
+1. Use data types with less computational workload. For example, if there is no obvious difference between the precision of the same operator in float16 and float32 modes, you can use the float16 format with less calculation workload.
+2. Use other operators with the same algorithm to avoid this problem.
+3. Pay attention to 16-alignment in the Ascend environment. Due to the design of the Ascend AI Processors, it is recommended that the calculation on the AI core be 16-alignment (each dimension in the shape is a multiple of 16).
+4. [Operator Tuning](https://mindspore.cn/tutorials/experts/en/master/debug/auto_tune.html).
+
+If you find an operator with poor performance, you are advised to contact [MindSpore community](https://gitee.com/mindspore/mindspore/issues) for feedback. We will optimize the operator in time after confirming that the problem is caused by poor performance.
+
+### Framework Enabling Performance Tuning
+
+#### Using the Static Graph Mode
+
+Generally, MindSpore in static graph mode is much faster than that in PyNative mode. It is recommended that training and inference be performed in static graph mode. For details, see [Combination of Dynamic and Static Graphs](https://www.mindspore.cn/docs/en/master/design/dynamic_graph_and_static_graph.html).
+
+#### On-device Execution
+
+MindSpore provides an [on-device execution method](https://www.mindspore.cn/docs/en/master/design/overview.html) to concurrently process data and execute the network on the device. You only need to set `dataset_sink_mode=True` in `model.train`. Note that this configuration is `True` by default. When this configuration is enabled, one epoch returns the result of only one network. You are advised to change the value to `False` during debugging.
+
+#### Using Automatic Mixed Precision
+
+The mixed precision training method accelerates the deep neural network training process by mixing the single-precision floating-point data format and the half-precision floating-point data format without compromising the network accuracy. Mixed precision training can accelerate the computing process, reduce memory usage and retrieval, and enable a larger model or batch size to be trained on specific hardware.
+
+For details, see [Mixed Precision Tutorial](https://www.mindspore.cn/tutorials/experts/en/master/others/mixed_precision.html).
+
+#### Enabling Graph Kernel Fusion
+
+Graph kernel fusion is a unique network performance optimization technology of MindSpore. It can automatically analyze and optimize the logic of existing network computational graphs, simplify and replace computational graphs, split and fuse operators, and build operators in a special way based on the target hardware capability to improve the computing resource utilization of devices and optimize the overall network performance. Compared with traditional optimization technologies, the graph kernel fusion technology has unique advantages, such as joint optimization of multiple operators across boundaries, cross-layer collaboration with operator compilation, and real-time compilation of operators based on Polyhedral. In addition, the entire optimization process of graph kernel fusion can be automatically completed after users enable the corresponding configuration. Network developers do not need to perform extra perception, so that users can focus on network algorithm implementation.
+
+Graph kernel fusion applies to scenarios that have high requirements on network execution time. Basic operators are combined to implement customized combination operators and these basic operators are automatically fused to improve the performance of the customized combination operators.
+
+For details, see [Graph Kernel Fusion Tutorial](https://www.mindspore.cn/docs/en/master/design/graph_fusion_engine.html).
+
+#### Others
+
+If there are too many conversion operators (TransData and Cast operators) and the conversion takes a long time, analyze the necessity of the manually added Cast operator. If the accuracy is not affected, delete the redundant Cast and TransData operators.
+
+If there are too many conversion operators automatically generated by MindSpore, the MindSpore framework may not be fully optimized for some special cases. In this case, contact [MindSpore community](https://gitee.com/mindspore/mindspore/issues) for feedback.
+
+In [dynamic shape scenario](https://www.mindspore.cn/docs/en/master/migration_guide/analysis_and_preparation.html), continuous graph build is required, which may cause a long end-to-end training time. You are advised to [avoid dynamic shape](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/model_and_loss.html).
+
+### Multi-Node Synchronization Performance Tuning
+
+During distributed training, after forward propagation and gradient calculation are complete in a step training process, each machine starts to perform AllReduce gradient synchronization. The AllReduce synchronization time is mainly affected by the number of weights and machines. For a more complex network with a larger machine scale, the AllReduce gradient update time is longer. In this case, you can perform AllReduce segmentation to reduce the time consumption.
+
+In normal cases, AllReduce gradient synchronization waits until all backward operators are executed. That is, after the gradient of all gradients is calculated, the gradients of all machines are synchronized at a time. After AllReduce segmentation is used, the gradients of some weights can be calculated, gradient synchronization of this part of weights is immediately performed. In this way, gradient synchronization and gradient calculation of remaining operators can be performed concurrently, and this part of AllReduce gradient synchronization time is hidden. The shard strategy is usually manually tried to find an optimal solution (more than two shards are supported).
+The [ResNet-50](https://gitee.com/mindspore/models/blob/master/official/cv/resnet/train.py) is used as an example. The network has 160 weights. [85, 160] indicates that gradient synchronization is performed immediately after the gradients of weights 0 to 85 are calculated, and gradient synchronization is performed after the gradients of weights 86 to 160 are calculated. The network is divided into two shards. Therefore, gradient synchronization needs to be performed twice. The sample code is as follows:
+
+```python
+import os
+import mindspore as ms
+from mindspore.communication import init
+
+device_id = int(os.getenv('DEVICE_ID', '0'))
+rank_size = int(os.getenv('RANK_SIZE', '1'))
+rank_id = int(os.getenv('RANK_ID', '0'))
+
+# init context
+ms.set_context(mode=ms.GRAPH_MODE, device_target='Ascend', device_id=device_id)
+if rank_size > 1:
+ ms.set_auto_parallel_context(device_num=rank_size, parallel_mode=ms.ParallelMode.DATA_PARALLEL,
+ gradients_mean=True)
+ ms.set_auto_parallel_context(all_reduce_fusion_config=[85, 160])
+ init()
+```
+
+For details, see [Cluster Performance Profiling](https://www.mindspore.cn/mindinsight/docs/en/master/performance_profiling_of_cluster.html).
+
+### Data Processing Performance Tuning
+
+The performance jitter of a single step and the empty data queue for a period of time are caused by the poor performance of the data preprocessing part. As a result, the data processing speed cannot keep up with the iteration speed of a single step. The two symptoms usually occur in pairs.
+
+When the data processing speed is slow, the empty queue is gradually consumed from the beginning when the queue is full. The training process starts to wait for the empty queue to fill in data. Once new data is filled in, the network continues single-step training. Because no queue is used as the buffer for data processing, the performance jitter of data processing is directly reflected by the performance of a single step. Therefore, the performance jitter of a single step is also caused.
+
+For details about data performance problems, see [Data Preparation Performance Analysis](https://www.mindspore.cn/mindinsight/docs/en/master/performance_profiling_ascend.html#data-preparation-performance-analysis) of MindInsight. This describes common data performance problems and solutions.
+
+For more performance debugging methods, see [Performance Tuning](https://www.mindspore.cn/tutorials/experts/en/master/debug/performance_optimization.html).
diff --git a/docs/mindspore/source_en/migration_guide/model_development/images/evaluation_procession.png b/docs/mindspore/source_en/migration_guide/model_development/images/evaluation_procession.png
new file mode 100644
index 0000000000000000000000000000000000000000..12392b142d1528f145d89f079b1860c2ca28b0ee
GIT binary patch
literal 23286
zcmeFZ2T)Vt`!yI*EFfY*P!Lc7>0qHKC8$W3-V9Zw*C+-MBm_}GMT%7EgceF55}Je(
z6#)Th2?VK8kP<>C)DW^a{^tLmoo{x&o&9EZXLjbhGt8ajl3U*LyyrR3IrpuJkq#FJ
zKL-c|;?mQ-X9fZt)B}MI_#8b9d~-f(Z2{OFcxt9|7gW)6fdYIu