[ONNX] Add external_data_dir backend parameter #3443
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
NNCF ONNX
alexsu52
reviewed
Apr 20, 2025
|
|
||
| external_data_dir = get_external_data_dir(advanced_parameters) | ||
| if external_data_dir: | ||
| model.metadata_props.add(key="nncf.external_data_dir", value=external_data_dir) |
There was a problem hiding this comment.
Will NNCF return a model with metadata_props nncf.external_data_dir?
There was a problem hiding this comment.
This is a good question. I will check. We probably shouldn't do that.
Adding this parameter to metadata_props is just one possible solution (the simplest one that requires minimal changes). We could also pass such a parameter explicitly.
andrey-churkin
force-pushed
the
ac/external_data_dir
branch
from
e48e7cf to
da2a08d
Compare
|
@alexsu52 Could you please take another look? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Added the
BackendParameters.EXTERNAL_DATA_DIRparameter for the ONNX backend. This parameter specifies the absolute path to the directory where the model’s external data files are stored. All external data files must be located in the same directory. It should be used when the model is loaded without external data usingonnx.load("model.onnx", load_external_data=False), and the external data files are not in the current working directory of the process. This parameter can be omitted if the external data files are located in the current working directory of the process.Added
check_model_protobuf_size()andcheck_external_data_location()validation checks. After this PR, callingnncf.quantize()on an ONNX model with a protobuf size exceeding 2GB will raise annncf.ValidationError. Thecheck_external_data_location()method ensures that the external data files can be successfully located.Reason for changes
Any proto in serialized form must be <2GB (Total Size of the Message)
To use
ort.InferenceSession(), we callmodel.SerializeToString()to pass the model as bytes. However, for large models (greater than 2GB), the serialized form also exceeds 2GB, which is not supported. As a result, passing such a model toort.InferenceSession()causes an error during inference.Related tickets
Ref: 164211
Tests
Current scope