OneFlow v1.0.0 came out, welcome to install the new version for a better experience.
Highlights
New Features
Improvements
Changes and Fixes
Performance
Highlights
This version update includes 447 commits and the following highlights:
Released a new interface compile_from_torch. This interface, while sharing the parameter memory, converts a PyTorch Module instance into a OneFlow Module instance. It supports direct Eager execution or conversion into a static graph nn.Graph, further accelerating the process using MLIR compilation. This interface is rapidly evolving and currently supports dynamic shape compilation, validated across typical models such as ResNet50, Faster RCNN, and Stable Diffusion.
Made a series of optimizations and refactoring to Eager execution runtime, including unification of system memory pools, integration with CUDA native interfaces, optimization of instruction scheduling mechanisms, introduction of an instruction fusion mechanism, optimization of Autograd graph construction speed, optimization of Op inference process, and decoupling of Instruction and Stream, etc.
The static graph distributed physical execution plan supports separate compilation functionality, allowing each process to independently compile its required execution plan, eliminating linear growth of compilation time with GPU scale.
Addition of a series of functional automatic differentiation related interface supports, including jvp, vjp, hvp, vhp, jacobian, and hessian.
Addition of the Insight module, supporting visualization of kernel invocation, execution time, speed, and other related information within the embedded point intervals.
Updates to LiBai (the open-source toolbox for large-scale model training), with native support for fine-tuning and distributed inference of Llama2 and ChatGLM2, supporting full finetune, adapter finetune, lora finetune. lm-eval-harness can be used for language model evaluation and validation.
Upgrade of OneFlow Serving functionality, adding support for OneFlow Python backend and OneFlow Lite backend, in addition to the existing support for OneFlow Cpp backend.
New Features
1. compile_from_torch
The compile_from_torch interface, while sharing the parameter memory, converts a PyTorch Module instance into a OneFlow Module instance. It supports direct Eager execution or conversion into a static graph nn.Graph, further accelerating the process using MLIR compilation. (https://github.com/Oneflow-Inc/oneflow/pull/10404, https://github.com/Oneflow-Inc/oneflow/pull/10408, https://github.com/Oneflow-Inc/oneflow/pull/9984, https://github.com/Oneflow-Inc/oneflow/pull/9754)
Interface Signature and Parameter Introduction:
compile_from_torch(torch_module: torch.nn.Module, \*, use_graph=True, options={})
* torch_module: The Torch Module instance to be converted.
* use_graph: Indicates whether to transform into a static graph nn.Graph and utilize MLIR compilation acceleration. The default is True.
* options:
* size: When using static graph nn.Graph, the hash value of the graph corresponding to the input shape will be calculated and cached. Size indicates the maximum capacity of the static graph cache. When exceeding the maximum capacity, the graph will be cleared based on the LRU strategy. The default value is 9.
* dynamic: For the first input with a dynamic shape, the graph will be fully compiled. For subsequent inputs with different shapes, if dynamic is True, shared graph will be used for compilation acceleration; if dynamic is False, the compilation will be performed each time. The default is True.
* debug: Debug mode and log level settings. -1 disables debug mode, 0 outputs warnings and static graph construction information, 1 additionally outputs graph construction information for each sub-module, 2 additionally outputs progress for each operator, 3 provides more detailed operator information. The default value is -1.
Example of Usage:
import torch
from torchvision import models
import oneflow
from oneflow.framework.infer_compiler import compile_from_torch
DEVICE = torch.device("cuda")
WEIGHT = models.ResNet50_Weights.DEFAULT
model = models.resnet50(weights=WEIGHT).to(DEVICE)
compile_model = compile_from_torch(model, options={"dynamic": True})
2. Separated Compilation
The static graph distributed physical execution plan supports separate compilation , allowing each process to independently compile its required execution plan, thereby preventing linear growth of compilation time with GPU scale. The separate compilation feature supports 3D hybrid parallel (data parallelism + model parallelism + pipeline parallelism) scenarios and can be used together with LiBai (the open-source large-scale model training toolbox). To enable this feature, use the command: export ONEFLOW_ENABLE_LAZY_SEPARATE_COMPILE=1. (https://github.com/Oneflow-Inc/oneflow/pull/9920, https://github.com/Oneflow-Inc/oneflow/pull/10140, https://github.com/Oneflow-Inc/oneflow/pull/10141, https://github.com/Oneflow-Inc/oneflow/pull/10124, https://github.com/Oneflow-Inc/oneflow/pull/10102)
Below are the test results on a 128-card A100-PCIE-40GB device with LiBai on the GPT2 model:
| Parallelism | Separated Compilation Enabled | Execution Plan Compilation Time |
| ------------ | ----------------------------------- | ------------------------------------ |
| Data Parallelism (DP128 MP1 PP1) | No | Over 20 minutes |
| Data Parallelism (DP128 MP1 PP1) | Yes | 108.21 s |
| 3D Parallelism (DP4 MP4 PP8) | No | 445.16 s |
| 3D Parallelism (DP4 MP4 PP8) | Yes | 82.88 s |
A series of functional automatic differentiation-related interfaces have been introduced, including jvp, vjp, hvp, vhp, jacobian, and hessian. (https://github.com/Oneflow-Inc/oneflow/pull/10412, https://github.com/Oneflow-Inc/oneflow/pull/10428)
Example of Usage:
import oneflow as flow
# jacobian example
def exp_reducer(x):
return x.exp().sum(dim=1)
input = flow.rand(2, 2)
jac_rslt = flow.autograd.functional.jacobian(exp_reducer, input)
# vhp example
def pow_reducer(x):
return x.pow(3).sum()
input = flow.rand(2, 2)
v = flow.ones(2, 2)
vhp_rslt = flow.autograd.functional.vhp(pow_reducer, input, v)
4. Insight Module
Introduced a new Insight module, enabling the visualization of kernel invocation, execution time, speed, and other related information within the embedded point intervals. (https://github.com/Oneflow-Inc/oneflow/pull/10370)
Usage:
Step 1: Set embedded point intervals in the code using the OneFlow Profiler module.
Step 2: Run the code and use NVIDIA Nsight Systems to generate a .sqlite file.
Step 3: Use the OneFlow Insight module to generate a .json file.
Step 4: Open the .json file in the browser at chrome://tracing/ or edge://tracing/ to obtain the visualization interface.
For more detailed information, please refer to: https://github.com/Oneflow-Inc/oneflow/tree/master/python/oneflow/utils/insight#usage
5. LiBai Version Update
LiBai (the open-source toolbox for large-scale model training) has been upgraded to version v0.3.0. It now natively supports finetuning and distributed inference of large language models Llama2 and ChatGLM2. It supports full full finetune, adapter finetune, lora finetune. lm-eval-harness can be used for language model evaluation and validation.
The distributed training and inference support for ChatGLM and Llama2 are as follows:
Added depend operator for describing dependency relationships in the computation graph. (https://github.com/Oneflow-Inc/oneflow/pull/9807)
Added operators for handling complex data types: real, imag, conj, conj_physical. (https://github.com/Oneflow-Inc/oneflow/pull/10034, https://github.com/Oneflow-Inc/oneflow/pull/10281)
Added CPU support for the nms operator. (https://github.com/Oneflow-Inc/oneflow/pull/10225)
Added support for the cast operator to convert bool to int16 data type. (https://github.com/Oneflow-Inc/oneflow/pull/10211)
Added support for the arange operator for the fp16 data type. (https://github.com/Oneflow-Inc/oneflow/pull/10019)
Added support for the adaptive_avg_pool operator for the fp16 data type. (https://github.com/Oneflow-Inc/oneflow/pull/10004)
Added support for the nonzero operator for the fp16 data type. (https://github.com/Oneflow-Inc/oneflow/pull/9826)
Added support for the exponential operator for the half data type. (https://github.com/Oneflow-Inc/oneflow/pull/10005)
Added support for the arg_sort and top_k operators for the half data type. (https://github.com/Oneflow-Inc/oneflow/pull/10000)
Added support for some basic operators like add, sub, mul, mm, sqrt, div for complex data types. (https://github.com/Oneflow-Inc/oneflow/pull/10269, https://github.com/Oneflow-Inc/oneflow/pull/10136, https://github.com/Oneflow-Inc/oneflow/pull/10284, https://github.com/Oneflow-Inc/oneflow/pull/10049)
Added support for basic binary operators for discontinuous memory input tensors. (https://github.com/Oneflow-Inc/oneflow/pull/9986)
Added a virtual jit interface to support mocking of torch for user code that imports but does not actually use the interface. (https://github.com/Oneflow-Inc/oneflow/pull/10395)
Added the mem_get_info interface to return overall and free memory information for a specified CUDA device. (https://github.com/Oneflow-Inc/oneflow/pull/10398)
Added the tensor.new interface. (https://github.com/Oneflow-Inc/oneflow/pull/9881)
Added the tensor.is_cpu interface. (https://github.com/Oneflow-Inc/oneflow/pull/10172)
Added the tensor.is_view interface. (https://github.com/Oneflow-Inc/oneflow/pull/10101)
Added the tensor.data_ptr interface. (https://github.com/Oneflow-Inc/oneflow/pull/10111, https://github.com/Oneflow-Inc/oneflow/pull/10139)
Added the tensor.baddbmm interface. (https://github.com/Oneflow-Inc/oneflow/pull/9918)
Added interfaces like special.erf, special.erfc, etc. (https://github.com/Oneflow-Inc/oneflow/pull/9982)
Added the layout and frombuffer interfaces. (https://github.com/Oneflow-Inc/oneflow/pull/10171)
Added the utils.model_zoo interface. (https://github.com/Oneflow-Inc/oneflow/pull/10183)
Added the get_rng_state and get_rng_state_all interfaces. (https://github.com/Oneflow-Inc/oneflow/pull/9760)
Added the set_rng_state and set_rng_state_all interfaces. (https://github.com/Oneflow-Inc/oneflow/pull/10250)
Added support for the float16 data type. (https://github.com/Oneflow-Inc/oneflow/pull/9697)
Added support for the char and short data types. (https://github.com/Oneflow-Inc/oneflow/pull/10086)
Added support for the complex64 and complex128 data types. (https://github.com/Oneflow-Inc/oneflow/pull/9987)
Integrated Transform Dialect into the MLIR codegen process. (https://github.com/Oneflow-Inc/oneflow/pull/10224, https://github.com/Oneflow-Inc/oneflow/pull/10227)
Added code generation support for the matmul operator. 。(https://github.com/Oneflow-Inc/oneflow/pull/10283)
Added code generation support for the softmax operator. (https://github.com/Oneflow-Inc/oneflow/pull/10263, https://github.com/Oneflow-Inc/oneflow/pull/10272)
Added code generation support for the transform.oneflow.apply_patterns operator. (https://github.com/Oneflow-Inc/oneflow/pull/10255)
Added support for byte attributes in the MLIR codegen process. (https://github.com/Oneflow-Inc/oneflow/pull/10276)
Added extra_libs functionality to the mock_torch module, enabling flowvision to mimic torchvision's functionality. (https://github.com/Oneflow-Inc/oneflow/pull/10223)
Added lazy parameter to the mock_torch module, allowing non-existent interfaces to return a fake object without immediate errors. (https://github.com/Oneflow-Inc/oneflow/pull/9876)
Added skip_init functionality and introduced meta device. (https://github.com/Oneflow-Inc/oneflow/pull/10008)
Introduced the HostMemoryInput mechanism, allowing an operator's specific input to be defined as HostMemoryInput type for accessing data within the kernel's host function body. (https://github.com/Oneflow-Inc/oneflow/pull/9928)
Added fusion mechanism for nccl logical operations to reduce excessive synchronization overhead in scenarios like ZERO, where too many fragmented nccl calls lead to significant training speed reduction. (https://github.com/Oneflow-Inc/oneflow/pull/9879)
Introduced a mechanism for re-computation of tensor operations. (https://github.com/Oneflow-Inc/oneflow/pull/9861)
Added support for backward_hook, register_full_backward_hook, and register_state_dict_pre_hook. (https://github.com/Oneflow-Inc/oneflow/pull/9837, https://github.com/Oneflow-Inc/oneflow/pull/9710)
Added content related to the stochastic weight averaging algorithm to the optimizers module. (https://github.com/Oneflow-Inc/oneflow/pull/9781)
Added the environment variable ONEFLOW_ENABLE_GLOBAL_INPUTS_WITH_INCONSISTENT_PLACEMENT to control whether to automatically place global tensors used by operators through the to_global operation on the largest rank. (https://github.com/Oneflow-Inc/oneflow/pull/10073)
Added the environment variable ONEFLOW_EAGER_NCCL_USE_COMPUTE_STREAM to control whether nccl and regular computations in eager mode are on the same stream. The default value is false. (https://github.com/Oneflow-Inc/oneflow/pull/10230)
Added the environment variable VLOG_REMAT to handle dynamic graph recomputation logs and interface with ComputeComplexityFn to estimate op computation time. (https://github.com/Oneflow-Inc/oneflow/pull/10212)
Added the environment variable ENABLE_ACTOR_DEBUG_LOG to print detailed logs of actor message sending, receiving, and execution on the current rank. (https://github.com/Oneflow-Inc/oneflow/pull/10081)
Added the environment variable ONEFLOW_RUN_GRAPH_BY_VM to control whether to use VM to run static graph nn.Graph. (https://github.com/Oneflow-Inc/oneflow/pull/9884)
Added the environment variable ONEFLOW_DISABLE_MOCK_TORCH to control whether to disable the mock_torch functionality. (https://github.com/Oneflow-Inc/oneflow/pull/9805)
Added the environment variable ONEFLOW_VM_MULTI_THREAD to control the number of threads used in the VM. (https://github.com/Oneflow-Inc/oneflow/pull/9698)
Added support for the second-order optimizer lbfgs. (https://github.com/Oneflow-Inc/oneflow/pull/10265)
Improvements
1. Eager Runtime Optimization and Refactoring
A series of optimizations and refactoring has been implemented for the Eager runtime, including:
Unified system memory pool to manage memory resources across all allocators on the same device. (https://github.com/Oneflow-Inc/oneflow/pull/8591)
Integration with CUDA native interfaces to accelerate kernel launches.(https://github.com/Oneflow-Inc/oneflow/pull/8571)
Optimization of instruction scheduling mechanism to reduce system overhead.(https://github.com/Oneflow-Inc/oneflow/pull/8796)
Introduction of instruction fusion mechanism to accelerate instruction dispatch. (https://github.com/Oneflow-Inc/oneflow/pull/7399)
Speed improvement in Autograd graph construction. (https://github.com/Oneflow-Inc/oneflow/pull/8606)
Optimization of op deduction process to accelerate kernel execution. (https://github.com/Oneflow-Inc/oneflow/pull/8672, https://github.com/Oneflow-Inc/oneflow/pull/8619, https://github.com/Oneflow-Inc/oneflow/pull/8662)
Consolidation of redundant concepts within the eager runtime, decoupling Instruction and Stream. (https://github.com/Oneflow-Inc/oneflow/pull/8583, https://github.com/Oneflow-Inc/oneflow/pull/8590, https://github.com/Oneflow-Inc/oneflow/pull/7607)
Users can configure the Eager runtime using various environment variables:
| Environment Variable | Meaning | Default Value |
| -------------------- | ------- | ------------- |
| ONEFLOW_VM_COMPUTE_ON_WORKER_THREAD | Whether to perform computation on worker threads | true |
| ONEFLOW_VM_MULTI_THREAD | Whether to use multi-threaded collaboration for Eager computation | true |
| ONEFLOW_VM_ENABLE_STREAM_WAIT | Whether to use stream_wait mechanism for dependencies between multiple streams | true |
| ONEFLOW_VM_ENABLE_SCHEDULE_YIELD | Whether to use yield mechanism to reduce scheduler thread's busy wait | true |
| ONEFLOW_EAGER_ENABLE_LOCAL_INFER_CACHE | Whether to cache operator output metadata during computation | true |
| ONEFLOW_VM_WORKER_THREAD_LIMIT | Number of worker threads | 16 |
| ONEFLOW_VM_PENDING_HANDLE_WINDOW_SIZE | Maximum size for fusing vm instructions | 10 |
| ONEFLOW_VM_BLOCKING_DEBUG_INSTRUCTIONS_DISPLAY_LIMIT | Number of unprocessed instructions to be printed when vm execution times out | 1000 |
2. Upgrade of OneFlow Serving Features
OneFlow Serving features have been upgraded to support additional backends, including OneFlow Python backend and OneFlow Lite backend, in addition to the existing support for the OneFlow Cpp backend.
The OneFlow Cpp backend enables deployment in a Python-independent environment to achieve the highest performance.
The OneFlow Lite backend enables deployment on edge devices.
The OneFlow Python backend facilitates the deployment of complex models with minimal migration cost.
For usage instructions, refer to: https://github.com/Oneflow-Inc/serving/blob/main/README.md
3. Other Functionality Improvements
Optimized certain code implementations to accommodate CUDA 12.x. (https://github.com/Oneflow-Inc/oneflow/pull/10367)
Optimized the glu operator implementation to support bias-less inputs.(https://github.com/Oneflow-Inc/oneflow/pull/9874)
Optimized pooling operator implementation to support the channels_last parameter. (https://github.com/Oneflow-Inc/oneflow/pull/10242)
Optimized the flip operator implementation to address memory access inefficiencies when dim = -1. (https://github.com/Oneflow-Inc/oneflow/pull/10310)
Optimized the bincount operator implementation for accelerated performance. (https://github.com/Oneflow-Inc/oneflow/pull/10308)
Optimized the index_add operator implementation by dispatching varied logic based on index length to enhance performance for smaller indices.(https://github.com/Oneflow-Inc/oneflow/pull/9751)
Optimized the topk operator implementation to boost performance when batch size equals 1. (https://github.com/Oneflow-Inc/oneflow/pull/10009)
Optimized implementations of operators such as conv and arange to facilitate CUDA graph usage. (https://github.com/Oneflow-Inc/oneflow/pull/9761)
Optimized the upsample operator implementation to include input/output size validation.(https://github.com/Oneflow-Inc/oneflow/pull/9737)
Optimized the grouped_matmul_bias operator implementation by introducing tensor parallelism sbp derivation rules. (https://github.com/Oneflow-Inc/oneflow/pull/9934)
Optimized the reshape operator implementation with added nd sbp derivation rules. (https://github.com/Oneflow-Inc/oneflow/pull/9858)
Optimized error messages and completed test cases for mask_fill and in_top_k operators. (https://github.com/Oneflow-Inc/oneflow/pull/10062)
Optimized the higher-order differentiation rules for the tanh operator to optimize performance under third-order differentiation. (https://github.com/Oneflow-Inc/oneflow/pull/10188, https://github.com/Oneflow-Inc/oneflow/pull/10237)
Optimized conv interface implementation to support device and dtype parameters. (https://github.com/Oneflow-Inc/oneflow/pull/10228)
Optimized conv interface implementation to automatically expand input dimensions.(https://github.com/Oneflow-Inc/oneflow/pull/9721)
Optimized sum interface implementation to accommodate dtype parameters.(https://github.com/Oneflow-Inc/oneflow/pull/10204)
Optimized softmax interface implementation to support dtype parameters. (https://github.com/Oneflow-Inc/oneflow/pull/10069)
Optimized maxpool interface implementation to support 3D input tensors. (https://github.com/Oneflow-Inc/oneflow/pull/10110)
Optimized ctc_loss interface implementation parameters with PyTorch interface. (https://github.com/Oneflow-Inc/oneflow/pull/9887)
Optimized copy interface implementation to support scenarios where input and output have different devices and dtypes. (https://github.com/Oneflow-Inc/oneflow/pull/9888)
Optimized grad interface implementation to support the allow_unused parameter.(https://github.com/Oneflow-Inc/oneflow/pull/10251)
Optimized load interface implementation to provide more user-friendly error messages.(https://github.com/Oneflow-Inc/oneflow/pull/10138)
Optimized fused_matmul_bias operator and interface implementation to support alpha and beta parameters. (https://github.com/Oneflow-Inc/oneflow/pull/10015)
Optimized normal operator and interface implementation to align behavior with PyTorch. (https://github.com/Oneflow-Inc/oneflow/pull/10185)
Optimized fused attention operator and interface implementation to allow None for pasti_key and past_value. (https://github.com/Oneflow-Inc/oneflow/pull/9977)
Optimized fused_attention operator and interface implementation to add support for variable sequence lengths. (https://github.com/Oneflow-Inc/oneflow/pull/9991)
Optimized fused_multi_head_attention_inference operator and interface implementation to include attn_bias parameter. (https://github.com/Oneflow-Inc/oneflow/pull/9853)
Optimized bn-related functor implementation. Merging bn_add_relu and bn_relu operations to expedite inference. (https://github.com/Oneflow-Inc/oneflow/pull/10239)
Optimized MLIR CodeGen-based processes and upgraded LLVM version to 16.0.0. (https://github.com/Oneflow-Inc/oneflow/pull/9985)
Optimized MLIR codegen-based processes by adding AppendOneFlowStream, MgpuToOneFlowStream, and CastOneFlowInputToSignlessPass passes. (https://github.com/Oneflow-Inc/oneflow/pull/10149, https://github.com/Oneflow-Inc/oneflow/pull/10151, https://github.com/Oneflow-Inc/oneflow/pull/10099)
Optimized MLIR codegen-based processes by linking LibDevice to support NVVM IR conversion to cubin. (https://github.com/Oneflow-Inc/oneflow/pull/10200)
Optimized MLIR codegen-based processes by utilizing tmpbuffer as MemPool in MLIR. (Oneflow-Inc/oneflow#10159)
Optimized MLIR codegen-based processes by enabling bufferizable operator dispatch. (https://github.com/Oneflow-Inc/oneflow/pull/9787)
Optimized MLIR codegen-based processes to expedite ofmempool and related processes. (https://github.com/Oneflow-Inc/oneflow/pull/10152, https://github.com/Oneflow-Inc/oneflow/pull/10168, https://github.com/Oneflow-Inc/oneflow/pull/10184, https://github.com/Oneflow-Inc/oneflow/pull/10239)
Optimized random number generator implementation by adding caching to avoid regeneration with each call. (https://github.com/Oneflow-Inc/oneflow/pull/10387)
Optimized graph load functionality to support loading the graph onto a new device.(https://github.com/Oneflow-Inc/oneflow/pull/10335)
Optimized dummy array initialization implementation using fold expressions. (https://github.com/Oneflow-Inc/oneflow/pull/10271)
Optimized MemoryFormat class organization, exposed to Python layer via cpython to support changing tensor's MemoryFormat using Tensor.to interface. (https://github.com/Oneflow-Inc/oneflow/pull/10181)
Optimized implementations of steam, device, and vm to support more device types. (https://github.com/Oneflow-Inc/oneflow/pull/10166)
Optimized error messages for MapAt, adding printing of key values.(https://github.com/Oneflow-Inc/oneflow/pull/10090)
Optimized OOM error messages to differentiate CUDA and CPU devices and display size. (https://github.com/Oneflow-Inc/oneflow/pull/9938)
Optimized error messages for CHECK_XX_OR_RETURN macros. (https://github.com/Oneflow-Inc/oneflow/pull/9921)
Optimized error messages for graph-related issues. (https://github.com/Oneflow-Inc/oneflow/pull/9821)
Optimized error messages for convolution operator-related issues. (https://github.com/Oneflow-Inc/oneflow/pull/9707)
Optimized model initialization to minimize additional overhead. (https://github.com/Oneflow-Inc/oneflow/pull/10088)
Optimized thread manager implementation to accommodate three usage scenarios: unrestricted threads, master as a thread, and n threads. (https://github.com/Oneflow-Inc/oneflow/pull/10060)
Optimized numpy array release mechanism to release in the main thread to reduce time-consuming GIL requests. (https://github.com/Oneflow-Inc/oneflow/pull/10050)
Optimized graph save runtime_state_dict implementation to enhance performance and address related issues. (https://github.com/Oneflow-Inc/oneflow/pull/10016)
Optimized parsing of different calling methods for interfaces like Tensor.foo(*args) using a unified PyParseArgs function. (https://github.com/Oneflow-Inc/oneflow/pull/9983)
Optimized the implementation of the ArgsTree class to support arbitrary output types and conducted file location migration. (https://github.com/Oneflow-Inc/oneflow/pull/9846)
Optimized memory allocation mechanism to achieve ordered allocation based on streams. (https://github.com/Oneflow-Inc/oneflow/pull/9818)
Removed debug compilation mode in graph compilation. (https://github.com/Oneflow-Inc/oneflow/pull/10145)
Removed unused logic for MemChain merge. (https://github.com/Oneflow-Inc/oneflow/pull/10097)
Removed default settings for some unused distributed environment variables. (https://github.com/Oneflow-Inc/oneflow/pull/9803)
Refactored collective boxing implementation under lazy mode. (https://github.com/Oneflow-Inc/oneflow/pull/10098)
Refactored registration of EagerCclS2S.(https://github.com/Oneflow-Inc/oneflow/pull/10100)
Refactored implementation of collective_boxing_executor_backend. (https://github.com/Oneflow-Inc/oneflow/pull/10082)
Refactored implementation of running global nn.graph using VM. (https://github.com/Oneflow-Inc/oneflow/pull/10048)
Refactored implementation of local to global related interfaces.(https://github.com/Oneflow-Inc/oneflow/pull/9870)
Refactored operator dispatch dialect implementation in MLIR codegen process. (https://github.com/Oneflow-Inc/oneflow/pull/9693)
Refactored implementation of random generator and distribution kernels. (https://github.com/Oneflow-Inc/oneflow/pull/9691)
Refactored implementation of fast_atomic_add operator. (https://github.com/Oneflow-Inc/oneflow/pull/9680)
Refactored error check related macros in glog. (https://github.com/Oneflow-Inc/oneflow/pull/10176)
Refactored implementation of random generator. (https://github.com/Oneflow-Inc/oneflow/pull/10025)
Refactored implementation of some elementwise primitive operations. (https://github.com/Oneflow-Inc/oneflow/pull/9857)
Refactored code related to device descriptions. (https://github.com/Oneflow-Inc/oneflow/pull/9791)
Refactored implementation of ParseDeviceString and ParseDeviceNameConf. (https://github.com/Oneflow-Inc/oneflow/pull/9833)
Refactored implementation of ActorMsg related functionalities, introducing IBVerbsActorMsgWrapper wrapper to reduce the size of ActorMsg. (https://github.com/Oneflow-Inc/oneflow/pull/9762)
Refactored implementation of save and load interfaces, migrating the method of saving graphs to the _save_graph function, adding some _open* helper classes to differentiate between paths and memory, enabling saving weights to BytesIO in save, and supporting file streaming in load. (https://github.com/Oneflow-Inc/oneflow/pull/10021)
Refactored implementation of some tensor-related interfaces, migrating code from Python layer to C++ layer. (https://github.com/Oneflow-Inc/oneflow/pull/9990, https://github.com/Oneflow-Inc/oneflow/pull/9964)
Upgraded PyBind version used in the project to 2.11.1. (https://github.com/Oneflow-Inc/oneflow/pull/10391)
2. Bug Fixes
Fixed default dynamic linking settings in CMake files to avoid LLVM15 linking errors. (https://github.com/Oneflow-Inc/oneflow/pull/10373, https://github.com/Oneflow-Inc/oneflow/pull/10131)
Fixed cast-related bugs in MLIR codegen. (https://github.com/Oneflow-Inc/oneflow/pull/10105)
Fixed logic handling for cpg attr in Module._apply function. (https://github.com/Oneflow-Inc/oneflow/pull/10343)
Fixed inheritance issue for DummyModule when attr is mro_entries. (https://github.com/Oneflow-Inc/oneflow/pull/9976)
Fixed size checking issue for _handle_size_arg in full op. (https://github.com/Oneflow-Inc/oneflow/pull/9975)
Fixed residual environment variables after launching mock via command line, causing subsequent API mock parameter errors. (https://github.com/Oneflow-Inc/oneflow/pull/9970)
Fixed inability to exit when two processes encounter exceptions. (https://github.com/Oneflow-Inc/oneflow/pull/10054)
Fixed bug in grouped quantization sbp derivation. (https://github.com/Oneflow-Inc/oneflow/pull/10132)
Fixed kMaxInputCount check issue in GroupedMatmulFunctor. (https://github.com/Oneflow-Inc/oneflow/pull/10322)
Fixed issue where double type attr was not updated when using shared_graph. (https://github.com/Oneflow-Inc/oneflow/pull/10279)
Fixed data type error in GetItemInScalarTensor. (https://github.com/Oneflow-Inc/oneflow/pull/10226)
Fixed gradient issue in GroupNorm, calling GroupNormParamGrad only when gamma and beta gradients are required. (https://github.com/Oneflow-Inc/oneflow/pull/10045)
Fixed error when reading tensors with partial ranks in global mode. (https://github.com/Oneflow-Inc/oneflow/pull/10056)
Fixed control boundary issues in checkpointing under PP, affecting task graph construction under separate compilation. (https://github.com/Oneflow-Inc/oneflow/pull/10057)
Fixed bug when using 3D parallelism and enabling activation checkpointing simultaneously. (https://github.com/Oneflow-Inc/oneflow/pull/10031)
Fixed adaptation bug of AutoMixedPrecision pass on non-CUDA devices and bug related to device combinations in LayerNorm Module. (https://github.com/Oneflow-Inc/oneflow/pull/10026)
Fixed default value setting issue for reduce parameter in scatter operator. (https://github.com/Oneflow-Inc/oneflow/pull/10002)
Fixed incomplete disable of some Torch variables in mock.disable, causing lingering references in other globals. (https://github.com/Oneflow-Inc/oneflow/pull/9989)
Fixed destructor issue in vm::TensorStorage. (https://github.com/Oneflow-Inc/oneflow/pull/9962)
Fixed offload issue where small tensors were not released from CUDA memory.(https://github.com/Oneflow-Inc/oneflow/pull/9974)
Fixed occasional segmentation fault in Python stack getter due to thread unsafety.(https://github.com/Oneflow-Inc/oneflow/pull/9955)
Fixed element lookup issue in set under separate compilation scenario. (https://github.com/Oneflow-Inc/oneflow/pull/9952)
Aligned qkv and output_layout in fused_multi_head_attention operator. (https://github.com/Oneflow-Inc/oneflow/pull/9950)
Fixed inconsistency in seed behavior of random series operators between graph and checkpointing. (https://github.com/Oneflow-Inc/oneflow/pull/9941)
Fixed parameter reload failure issue in Eager mode. (https://github.com/Oneflow-Inc/oneflow/pull/9935)
Fixed infinite loop issue in specific cases of mock torch lazy functionality. (https://github.com/Oneflow-Inc/oneflow/pull/9926)
Fixed issue where code in stft_kernel.cu file was not compiled by default. (Oneflow-Inc/oneflow#9922)
Fixed deadlock and memory allocation errors caused by invalid topological order due to incomplete TaskGraph under separate compilation in order_in_graph. (https://github.com/Oneflow-Inc/oneflow/pull/9909 )
Fixed xrt compilation issue where fmt could not be found. (https://github.com/Oneflow-Inc/oneflow/pull/9894)
Fixed imbalance in GPU memory allocation among processes during local to global process where sbp is B. (https://github.com/Oneflow-Inc/oneflow/pull/9852)
Aligned OneFlow and PyTorch behaviors related to the third parameter of CTCLoss. (https://github.com/Oneflow-Inc/oneflow/pull/9845)
Fixed initialization issues related to thread_global_id and rank_group_scope. (https://github.com/Oneflow-Inc/oneflow/pull/9841)
Fixed inplace handling errors in dropout operator implementation. (https://github.com/Oneflow-Inc/oneflow/pull/9808)
Fixed errors in loading non-tensor objects saved by PyTorch in the load function. (https://github.com/Oneflow-Inc/oneflow/pull/9804)
Fixed conflicts between contiguous memory and GPU memory allocation strategies. (https://github.com/Oneflow-Inc/oneflow/pull/9786)
Fixed memory allocation issues in EagerBlobObject::ByteSizeOfBlobBody when considering non-contiguous cases. (https://github.com/Oneflow-Inc/oneflow/pull/9782)
Fixed dtype inference errors in fill_ operator during autocast. (https://github.com/Oneflow-Inc/oneflow/pull/9776)
Fixed sbp derivation rule issues in fused_glu operator. (https://github.com/Oneflow-Inc/oneflow/pull/10108)
Fixed issues related to calling nn.Graph.__map_io. (https://github.com/Oneflow-Inc/oneflow/pull/10084)
Fixed inconsistency between set_grad_mode interface and PyTorch behavior. (https://github.com/Oneflow-Inc/oneflow/pull/10059)
Fixed an issue related to the map_location parameter in the load interface and added support for passing lambda functions. (https://github.com/Oneflow-Inc/oneflow/pull/10052)
Fixed stride inference errors after unsqueeze operation in view mode. (https://github.com/Oneflow-Inc/oneflow/pull/9775)
Fixed problems in conv op with unbatched input and bias, and added support for unbatched input in deconv op. (https://github.com/Oneflow-Inc/oneflow/pull/9740)
Fixed logic errors in trunc_normal_ implementation. (https://github.com/Oneflow-Inc/oneflow/pull/9711)
Fixed default value issue in dim parameter of topk operator. (https://github.com/Oneflow-Inc/oneflow/pull/9703)
Fixed issues where placement of some networks was incorrectly set to CPU during static graph printing. (https://github.com/Oneflow-Inc/oneflow/pull/9770)
Fixed conflict between include paths of trt_flash_attention and native flash attention. (https://github.com/Oneflow-Inc/oneflow/pull/9750)
Fixed segmentation fault caused by is_shutting_down and gil in stack getter. (https://github.com/Oneflow-Inc/oneflow/pull/9681)
Fixed issues related to the separate compilation feature found in distributed unit testing.(https://github.com/Oneflow-Inc/oneflow/pull/9749)
Fixed memory handling issues in flatten algorithm implementation. (https://github.com/Oneflow-Inc/oneflow/pull/9746)
Fixed a deadlock issue in the execution flow. (https://github.com/Oneflow-Inc/oneflow/pull/9738)
Fixed errors in isinstance check for DummyModule. (https://github.com/Oneflow-Inc/oneflow/pull/10207)
Corrected behavior where default size was erroneously overridden when introducing llvm::SmallVector. (https://github.com/Oneflow-Inc/oneflow/pull/9932)
Fixed errors in calculating memory size of non-contiguous memory tensors. (https://github.com/Oneflow-Inc/oneflow/pull/9819)
Fixed issues with calling CHECK_JUST in the TensorStorage destructor function. (https://github.com/Oneflow-Inc/oneflow/pull/9752)
Performance
1. OneFlow compile_from_torch VS PyTorch compile
Compile and execute the backbone parts of ResNet50 and Faster RCNN models using OneFlow compile_from_torch and PyTorch compile interfaces to test the compilation time with inputs of different shapes. The results are shown in the table below:
| Model | input shape | PyTorch compile | OneFlow compile_from_torch | dynamic | test timing |
| ---- | ---------- | --------------- | ---------------------------- | ------- | -------- |
| ResNet50 | (1, 3, 512, 512) | 21.328 s | 3.205 s | False | initial compilation and execution |
| ResNet50 | (2, 3, 896, 512) | 14.167 s | 1.523 s | False | continuous compilation and execution |
| ResNet50 | (2, 3, 512, 896) | 13.364 s | 1.402 s | False | continuous compilation and execution |
| ResNet50 | (3, 3, 896, 896) | 15.056 s | 1.539 s | False | continuous compilation and execution |
| ResNet50 | (2, 3, 1024, 896) | 14.167 s | 1.500 s | False | continuous compilation and execution |
| ResNet50 | (2, 3, 896, 1024) | 12.891 s | 1.494 s | False | continuous compilation and execution |
| ResNet50 | (6, 3, 1024, 1024) | 14.859 s | 1.872 s | False | continuous compilation and execution |
| ResNet50 | (1, 3, 512, 512) | 170.446 s | 3.143 s | True | initial compilation and execution |
| ResNet50 | (2, 3, 896, 512) | 185.672 s | 0.851 s | True | continuous compilation and execution |
| ResNet50 | (2, 3, 512, 896) | 0.089 s | 0.836 s | True | continuous compilation and execution |
| ResNet50 | (3, 3, 896, 896) | 0.084 s | 0.980 s | True | continuous compilation and execution |
| ResNet50 | (2, 3, 1024, 896) | 0.077 s | 0.942 s | True | continuous compilation and execution |
| ResNet50 | (2, 3, 896, 1024) | 0.080 s | 0.931 s | True | continuous compilation and execution |
| ResNet50 | (6, 3, 1024, 1024) | 0.084 s | 1.406 s | True | continuous compilation and execution |
| Faster RCNN | (1, 3, 512, 512) | 18.224 s | 5.483 s | False | initial compilation and execution |
| Faster RCNN | (2, 3, 896, 512) | 9.200 s | 3.011 s | False | continuous compilation and execution |
| Faster RCNN | (2, 3, 512, 896) | 9.331 s | 3.025 s | False | continuous compilation and execution |
| Faster RCNN | (3, 3, 896, 896) | 9.301 s | 2.854 s | False | continuous compilation and execution |
| Faster RCNN | (2, 3, 1024, 896) | 9.290 s | 2.805 s | False | continuous compilation and execution |
| Faster RCNN | (2, 3, 896, 1024) | 9.123 s | 2.851 s | False | continuous compilation and execution |
| Faster RCNN | (6, 3, 1024, 1024) | 9.377 s | 3.180 s | False | continuous compilation and execution |
| Faster RCNN | (1, 3, 512, 512) | 25.444 s | 5.430 s | True | initial compilation and execution |
| Faster RCNN | (2, 3, 896, 512) | 25.381 s | 1.899 s | True | continuous compilation and execution |
| Faster RCNN | (2, 3, 512, 896) | 0.116 s | 1.886 s | True | continuous compilation and execution |
| Faster RCNN | (3, 3, 896, 896) | 1.982 s | 1.793 s | True | continuous compilation and execution |
| Faster RCNN | (2, 3, 1024, 896) | 0.114 s | 1.803 s | True | continuous compilation and execution |
| Faster RCNN | (2, 3, 896, 1024) | 0.111 s | 1.778 s | True | continuous compilation and execution |
| Faster RCNN | (6, 3, 1024, 1024) | 0.143 s | 2.110 s | True | continuous compilation and execution |
Using the OneFlow compile_from_torch and PyTorch compile interfaces, the unet section of the Stable Diffusion model was compiled and executed to test the compilation time and execution time with outputs of different shapes. The results are presented in the table below:
| Model | Output shape | PyTorch compile | OneFlow compile_from_torch | dynamic | test timing |
| ---- | ---------- | --------------- | ---------------------------- | ------- | -------- |
| Stable Diffusion | (2, 512, 512) | 103.701 s | 63.670 s | False | initial compilation and execution |
| Stable Diffusion | (1, 512, 768) | 95.137 s | 53.864 s | False | continuous compilation and execution |
| Stable Diffusion | (2, 768, 512) | 90.259 s | 55.271 s | False | continuous compilation and execution |
| Stable Diffusion | (1, 768, 768) | 90.196 s | 51.590 s | False | continuous compilation and execution |
| Stable Diffusion | (2, 512, 512) | 275.660 s | 57.117 s | True | initial compilation and execution |
| Stable Diffusion | (1, 512, 768) | 345.774 s | 43.752 s | True | continuous compilation and execution |
| Stable Diffusion | (2, 768, 512) | 349.835 s | 47.653 s | True | continuous compilation and execution |
| Stable Diffusion | (1, 768, 768) | 7.224 s | 45.720 s | True | continuous compilation and execution |
| Stable Diffusion | (2, 512, 512) | 4.088 s | 2.831 s | False | subsequent execution |
| Stable Diffusion | (1, 512, 768) | 3.296 s | 2.325 s | False | subsequent execution |
| Stable Diffusion | (2, 768, 512) | 5.594 s | 5.157 s | False | subsequent execution |
| Stable Diffusion | (1, 768, 768) | 4.713 s | 3.557 s | False | subsequent execution |
| Stable Diffusion | (2, 512, 512) | 4.448 s | 2.801 s | True | subsequent execution |
| Stable Diffusion | (1, 512, 768) | 3.201 s | 2.314 s | True | subsequent execution |
| Stable Diffusion | (2, 768, 512) | 6.093 s | 4.166 s | True | subsequent execution |
| Stable Diffusion | (1, 768, 768) | 4.920 s | 3.557 s | True | subsequent execution |
Conclusion: The OneFlow compile_from_torch interface generally has shorter compilation times compared to the PyTorch compile interface. Additionally, benefiting from the exceptional operator optimizations in the OneFlow framework, there is superior execution performance on the Stable Diffusion model.
Note: The tests were conducted with GPU 3090, PyTorch v2.1.2 and CUDA 12.2.
Conclusion: Compared to PyTorch Eager, using OneFlow Eager shows significant performance advantages in small batch scenarios for both ResNet50 and BERT models.
Note: The tests were conducted using PyTorch v2.1.0 and CUDA 12.1.