bksahu’s GSoC warp up
Hello everyone!
GSoC 2019 has almost come to an end! It’s the time to wrap up this mega event started back in May 2019. Under the mentorship of Mentor Hayen, my learning experience has undergone a roller-coaster ride and it has not only boosted my growth as a developer but also as an individual. Over the last 3 months the following are my major contributions to this project:
Built-ins Optimizations
- “any”: PR #246
nuitka.nodes.BuiltinAnyNodes node added to optimize the “any” built-in.
Developed an algorithm to predict the “any” for arguments having repetitive values at compile time. For example:
any([0]*255) -> False
any([False, False, True]) -> True
Extended support for range, set and dict built-ins.
Added the optimized C side support too
Added a method getMetaClassBase to make Python 2 and Python 3 compatible while working with metaclasses.
- Issue reported and closed Issue #349
Created a new module nuitka.nodes.IterationHandles to work with iterables.
Added support of Iteration for non-mutable types.
- “all”: PR #407
Added nuitka.nodes.BuiltinAllNodes to optimize the “all” built-ins.
Developed an algorithm similar to “any” to predict the “all” arguments. For example:
all([0, 0, 1]) -> False
all([True]*100) -> True
Other similar optimizations are done like “any” built-in.
Additionally, added a new testing module tests.optimizations.CommonOptimizations to test the built-ins optimizations at the same place.
- “abs”: PR #419
Added new operation node ExpressionOperationAbs to optimize the abs built-in.
Manually added shapeSlotAbs to different shapes.
Finally pre-computed the compile time constant abs
- “max” and “min”: PR #442
This PR is work in progress and is half complete.
This is the first optimizations in which I used reformulations instead of added in a new node.
Pseudo-code of “min” reformulation:
def _min(a, b, c, ...): tmp_arg1 = a tmp_arg2 = b tmp_arg3 = c ... result = tmp_arg1 if keyfunc is None: # can be decided during re-formulation tmp_key_result = keyfunc(result) tmp_key_candidate = keyfunc(tmp_arg2) if tmp_key_candidate < tmp_key_result: result = tmp_arg2 tmp_key_result = tmp_key_candidate tmp_key_candidate = keyfunc(tmp_arg3) if tmp_key_candidate < tmp_key_result: result = tmp_arg3 tmp_key_result = tmp_key_candidate ... else: if tmp_arg2 < result: result = tmp_arg2 if tmp_arg3 < result: result = tmp_arg3 ... return result
Adding support for keyfunc is pending
- “zip”: PR #462
This built-in uses both types of optimizations that the previous built-ins optimizations used.
zip for Python 2 uses the reformulations.
Pseudo-code of “zip” reformulation:
def _zip(a, b, c, ... ): # First assign, to preserve order of execution, # the arguments might be complex expressions. tmp_arg1 = a tmp_arg2 = b tmp_arg3 = c ... tmp_iter_1 = iter(tmp_arg1) tmp_iter_2 = iter(tmp_arg2) tmp_iter_3 = iter(tmp_arg3) ... # could be more tmp_result = [] try: while 1: tmp_result.append( ( next(tmp_iter_1), next(tmp_iter_2), next(tmp_iter_3), ... ) ) except StopIteration: pass return tmp_result
zip for Python 3 needs a new node that calls the zip because unlike zip in Python 2, zip in Python 3 returns a zipobject.
Test suite
- Search mode “All”: PR #378
In the test suite, I added a new search mode “all” that will test all the modules and return all the results at once. For example:
$ ./tests/basics/run_all.py all Using concrete python 2.7.12 on x86_64 Comparing output of 'Asserts.py' using '/usr/bin/python' with flags silent, expect_success, remove_output, recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ... . . . . Total 0 error(s) found.
- Search mode “Only”: PR #333
Added “only” search mode to test only a single module. For example:
$ ./tests/basics/run_all.py only BuiltinsTest.py Using concrete python 2.7.12 on x86_64 Skipping Asserts.py Skipping Assignments.py Skipping BigConstants.py Skipping Branching.py Skipping BuiltinOverload.py Skipping BuiltinSuper.py Comparing output of 'BuiltinsTest.py' using '/usr/bin/python' with flags silent, expect_success, remove_output, recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
- Reported and closed Issue #334: PR #336
Fixed the autoformat bug by reading and writing the files in bytes instead of string.
Documentation
Nuitka git work flow: PR #485
And other minor doc fixes are added with their respective pull requests.
What I learned
Learned the software engineering principles and how to keep my work clean.
I also learned how to effectively use software designing principles like DRY and KISS.
Got exposed to Nuitka internals which helped me to better understand how compilers in general work.
Explored how CPython works internally.
Got some great advice from Mentor Hayen about starting my professional career in Software engineering.
Overall, it was a great experience to be a part of Nuitka :)