Skip to content

chore(performance): Use simd for base64 encoding/decoding #1379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

JakubOnderka
Copy link
Contributor

Summary

Use SIMD instructions when doing base64 encoding/decoding. base64 encoding/decoding is common when processing logs, so it would be nice to use SIMD powered algo. base64-simd crate is already used by vector.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

Standard tests.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on
    our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

  • Our CONTRIBUTING.md is a good starting place.
  • If this PR introduces changes to LICENSE-3rdparty.csv, please
    run dd-rust-license-tool write and commit the changes. More details here.
  • For new VRL functions, please also create a sibling PR in Vector to document the new function.

@JakubOnderka JakubOnderka force-pushed the base64-simd branch 3 times, most recently from b98a050 to c5ce9f5 Compare April 14, 2025 09:35
@pront
Copy link
Member

pront commented Apr 14, 2025

Hi @JakubOnderka, did you get compare numbers for this?

@JakubOnderka
Copy link
Contributor Author

% cargo bench --all-features base64
vrl_stdlib/functions/decode_base64/literal
                        time:   [55.005 ns 55.128 ns 55.258 ns]
                        thrpt:  [18.097 Melem/s 18.140 Melem/s 18.180 Melem/s]
                 change:
                        time:   [-30.458% -30.269% -30.095%] (p = 0.00 < 0.05)
                        thrpt:  [+43.051% +43.407% +43.798%]
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

vrl_stdlib/functions/encode_base64/literal
                        time:   [48.520 ns 48.602 ns 48.696 ns]
                        thrpt:  [20.535 Melem/s 20.575 Melem/s 20.610 Melem/s]
                 change:
                        time:   [-32.638% -32.400% -32.112%] (p = 0.00 < 0.05)
                        thrpt:  [+47.302% +47.930% +48.451%]
                        Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
  9 (9.00%) high mild
  8 (8.00%) high severe

Measured on Apple M2. Results can differ for x86_64 as it supports different SIMD instructions set. Also the benchmark encode/decode very short text, for longer strings it will be even faster.

@pront pront self-assigned this Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants