Skip to content

Use make instead of append to reduce memory usage #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vincent-163
Copy link

Since the pointers to original values are always held, it doesn't make sense to use append() and keep the original values anyway.
I expect this change to reduce memory consumption by about 30% for large JSON values and avoid copying.

@codecov
Copy link

codecov bot commented Jan 19, 2019

Codecov Report

Merging #23 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #23      +/-   ##
=========================================
+ Coverage   92.78%   92.8%   +0.02%     
=========================================
  Files           8       8              
  Lines         970     973       +3     
=========================================
+ Hits          900     903       +3     
  Misses         48      48              
  Partials       22      22
Impacted Files Coverage Δ
parser.go 90.37% <100%> (+0.06%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1ad9dd...74f997f. Read the comment docs.

@@ -64,7 +64,11 @@ func (c *cache) getValue() *Value {
if cap(c.vs) > len(c.vs) {
c.vs = c.vs[:len(c.vs)+1]
} else {
c.vs = append(c.vs, Value{})
if len(c.vs) == 0 {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

append already does something similar underneath - see https://play.golang.org/p/wFzqBZvEXPl . If append works slower than this code, then a bug must be filled at https://github.com/golang/go/issues

@valyala
Copy link
Owner

valyala commented Jan 19, 2019

I expect this change to reduce memory consumption by about 30% for large JSON values and avoid copying.

Uh I didn't notice this line. I thought the patch is for performance optimization, not a memory optimization. Then it looks valid. Could you add a benchmark with ReportAllocs or reference an existing benchmark, which shows the claimed improvement in memory usage?

@vincent-163
Copy link
Author

In fact, this will also improve performance by avoiding the copy. The benchmarks are here: https://gist.github.com/hewenyang/91618e54342f65809f4cb3c9c3aa1240
The observation is based on the canada benchmark which has the largest memory consumption. Its memory consumption went down from 1019612 B/op to 707650 B/op, and all parse benchmarks had a small but visible improvement in performance.

@harshavardhana
Copy link

@valyala can this be merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants