New feature: Local distribution capability #129
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfeiwong
reviewed
Oct 12, 2024
sfeiwong
reviewed
Oct 12, 2024
|
I changed some code to make it more concise Key changes:
|
HuHeng
reviewed
Oct 29, 2024
HuHeng
approved these changes
Oct 30, 2024
HuHeng
reviewed
Oct 30, 2024
update Makefile for c_engine test codes complete test/c_module configure test/cpp_builder Added untracked files add test code about data splitting add Makefile for bmf/sdk/cpp_sdk/test update Makefile and task.cpp refactor code for multi node test 1.add future that bmf::builder::Graph::Decode and Encode function can set schedule through initing 2.optimize the code of test/multi_node for test Supplement the last unsubmitted code delete useless code add feature that support output stream push data to multi downstream node add feature that InputStreamManager can handle multi inputs from multi upstreams optimize code fix bug that two node's OSM use one static variable add develop doc update fix bug that drop some frame from InputStream fix bug that encoder couldn't handle multi video inputstreams add pictures that missing update doc_dev/README.md 1.add assemble_module 2.complete function for interl node init 1.complete assemble_module 2.update doc_dev/README.md optimizer graph config if there is multi-thread node 1.Improve the optimizer processing of node multi-thread parameters 2.Implement data assemble in assemble_module 3.Implement OutputStreamManager's parent and subclass methods separately Fixed the bug that the number of distributed node threads cannot exceed 3 1. Added split module and improved optimizer 1. Add split and aggregation modules to the engine 2. Delete some useless implementations remove useless files remove useless files revert some unnecessary changes * rename test/multi_node to test/distributed_node * update enhance_demo.py * update enhance_module.py * update test_video_c_module.py resolve conflict restore builder.cpp and builder.hpp and cmakelist restore input_stream_manager and builder.cpp
* rename paramater "thread" to "dist_nums" * optimize process_distributed_node()
* remove SplitOutputStreamManager * optimize distributed node test case
* improve the code of optimizer * change the BMFLOG_NODE type of split and assemble module
* adjust some code lines in optimizer.cpp
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm excited to introduce a new feature that enables local distribution capability for BMF, allowing users to easily
parallelize computations on a single node.
In this commit, I've implemented the necessary code changes to support multi-threading for single node. To enable this
feature, users can simply add an option parameter
"thread":<number>when setting up these nodes, and the framework will create multiple distributed nodes to process data in parallel.Key Changes: