Skip to content

[BUG] Gemma3 Support in MLXModel #974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tanliboy opened this issue Mar 14, 2025 · 2 comments · May be fixed by #1074
Open

[BUG] Gemma3 Support in MLXModel #974

tanliboy opened this issue Mar 14, 2025 · 2 comments · May be fixed by #1074
Labels
bug Something isn't working

Comments

@tanliboy
Copy link

Describe the bug
MLXModel doesn't support Gemm3

Code to reproduce the error

➜  agent git:(agent) ✗ python3
Python 3.10.16 (main, Mar  4 2025, 12:34:35) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from smolagents import MLXModel
Python-dotenv could not parse statement starting at line 33
>>> engine = MLXModel(model_id="mlx-community/gemma-3-4b-it-8bit",max_tokens=10000)
Fetching 10 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 169125.16it/s]
ERROR:root:Model type gemma3 not supported.
Traceback (most recent call last):
  File "/Users/litan/.pyenv/versions/3.10.16/envs/shoppingwise/lib/python3.10/site-packages/mlx_lm/utils.py", line 148, in _get_classes
    arch = importlib.import_module(f"mlx_lm.models.{model_type}")
  File "/Users/litan/.pyenv/versions/3.10.16/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mlx_lm.models.gemma3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/litan/.pyenv/versions/3.10.16/envs/shoppingwise/lib/python3.10/site-packages/smolagents/models.py", line 522, in __init__
    self.model, self.tokenizer = mlx_lm.load(model_id, tokenizer_config={"trust_remote_code": trust_remote_code})
  File "/Users/litan/.pyenv/versions/3.10.16/envs/shoppingwise/lib/python3.10/site-packages/mlx_lm/utils.py", line 785, in load
    model, config = load_model(model_path, lazy)
  File "/Users/litan/.pyenv/versions/3.10.16/envs/shoppingwise/lib/python3.10/site-packages/mlx_lm/utils.py", line 720, in load_model
    model_class, model_args_class = get_model_classes(config=config)
  File "/Users/litan/.pyenv/versions/3.10.16/envs/shoppingwise/lib/python3.10/site-packages/mlx_lm/utils.py", line 152, in _get_classes
    raise ValueError(msg)
ValueError: Model type gemma3 not supported.
>>> engine = MLXModel(model_id="mlx-community/gemma-3-4b-it-8bit",max_tokens=10000)

Error logs (if any)
See the command line output above.

Additional context
I can get the output through python -m mlx_vlm.generate --model mlx-community/gemma-3-4b-it-8bit --max-tokens 100 --temperature 0.0 --prompt "How to make sushi?" and I assume the problem was more on the smolagents' usage.

Packages version:
Run pip freeze | grep smolagents and paste it here.

agent git:(agent) ✗ pip freeze | grep smolagents
openinference-instrumentation-smolagents==0.1.6
smolagents==1.10

Additional context
Add any other context about the problem here.

@tanliboy tanliboy added the bug Something isn't working label Mar 14, 2025
@tanliboy
Copy link
Author

It seems that we need to use mlx_vlm instead mlx_llm.

It there a way that we can make it compatible with the VLM?

@SauravMaheshkar SauravMaheshkar linked a pull request Mar 25, 2025 that will close this issue
@SauravMaheshkar
Copy link
Contributor

Hey @tanliboy I made a PR with a MLXVLMModel class in #1074, hope it helps. It's not merged yet but it's a few lines of code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants