Add librt functionality for lazy deserialization#21158
Add librt functionality for lazy deserialization#21158ilevkivskyi merged 7 commits intopython:masterfrom
Conversation
|
Hm, IIUC I should bump I guess we should use the local version of |
This comment has been minimized.
This comment has been minimized.
|
This also makes me think that if anyone is using compiled mypy form master, it will break as soon as I merge this. So maybe a better approach is to actually release this in |
Makes sense. |
| return _skip_int(data); | ||
| if (tag == INSTANCE) | ||
| return _skip_instance(data); | ||
| if (tag > MYPY_FILE && tag < RESERVED) |
There was a problem hiding this comment.
This is actually intentional, a little sanity check. We know that we should only read symbol nodes using this function, and MypyFile should be always handled using cross_ref in cache, and never appear explicitly in symbol tables.
| Py_ssize_t size = _read_size(data); | ||
| if (size < 0) | ||
| return CPY_NONE_ERROR; | ||
| int i; |
There was a problem hiding this comment.
Should this have type Py_ssize_t? (Here and other similar functions.)
There was a problem hiding this comment.
Yeah, I guess I just copied this from somewhere and all test passed. I will check what is the best cross-platform type here.
This comment has been minimized.
This comment has been minimized.
See python/mypy#21158 for motivation of non-master sync.
Together with #21158, this gets us 95% there for lazy desrialization. Implementation is generally straightforward. At some point I wanted to unify the new global state with (parts of) the checker state, and potentially get rid of a bunch of `named_type` callbacks we pass around, but decided to do this later in a separate PR, since this is not strictly necessary for parallel checking. Although this PR itself only makes fixup phase (which is the smaller part of deserialization) lazy, it already shows small performance improvements: * ~net zero on self-check * few percent faster cold `torch` with 4 workers * ~15% faster warm `torch`
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
2 participants
I am going to add actual lazy deserialization in a separate PR, probably next week. It will require one more preparatory PR setting up global state for
lookup_fully_qualified()that I will probably add on the weekend.cc @JukkaL