Sure those are good designs... But Lucen is not about "here's a nicer way to write loops," it's "you shouldn't have to touch the loop you already have."..... showing Lucen parallelize the ugly version is the point.
This code seems to be fairly pervasively AI-written (both structurally/syntacticly, with many additional AI verbosity tells in the documentation, and going by the commit history).
That's a problem for me, given what this library does: arbitrary transformations of Python code at load time. The fragility, security risk, and complexity added by basically writing a subdialect of Python with new semantics is something I'd like to see more human attention on and maturity of before I consider using it. There are times when AI smell doesn't concern me much when deciding whether or not to use someone's code. This isn't one of those times.
#pragma is a real keyword in C but # is just a comment in python - why not use a pythonic @decorator?
Vendoring a library like this into an app seems like a lot more toil than just writing the native multiprocessing python code, or using something built for number crunching like arrayfire or numpy
Because (even if you could decorate for-loops, or instead decorated functions containing a loop and nothing else) the resulting behavior would be completely unlike the behavior of all other @decorators in Python. It would also be nearly impossible to implement this functionality as a decorator without introducing tons of fragility.
The preprocessor/import-hook hack that allows macro-like behavior in Python does arbitrary transforms at the text/code-load level, while decorators are evaluated at runtime and run regular Python code. Decorators can be composed/wrapped by other functions and decorators, whereas implementing Lucen-like functionality with a decorator would necessarily match on the name/symbol of the decorator and couldn't indirect "through" it.
Heck, even if you disallowed wrapping and matched on the decorator name during preprocessing (basically stripping it out and rewriting decorated code strings at load time), even identifying the correct decorator as a transformation target would require reimplementing a ton of Python's import/scope/name resolution behavior by hand at code-transformation time. If your decorator was called lucen.parallelize(), consider the difference between "import lucen; @lucen.parallelize", "from lucen import parallelize; @parallelize", "from lucen import parallelize as pl; parallelize = 123; @pl", and so on. You'd have to handle all of those cases, and many more (e.g. decorated functions defined inside other classes/functions) at the code-as-string or AST level. You couldn't run the decorator at runtime, because a) syntactic information is gone by then, and b) because then you'd need to unimport/reimport module code which had already been run with arbitrary global side effects.
More information on the methods available to get preprocessor-like functionality in Python:
Import hooks can be configured at runtime, and run during subsequent import statements. They were originally designed to allow customized module discovery, but nothing stops you from using them like Lucen does: to intercept imports of local Python files and transform/replace the code imported at load time: https://peps.python.org/pep-0302/
just wanted to make sure ... even if someone someday wishes to revert the decision of using lucen they will be okay with just removing the import hooks - and run everything as native -- and when they have time, they can clear the comments but till then they can work without thinking
Interesting to see the rust code is doing the actual parallelization, while execution. Will it work for complex loops, and what happens if the loops uses global context and different variables, outside of the loop?. May be it should go to python standard library.
Not very Pythonic
Better Best Raymond Hettinger (Python core dev): https://gist.github.com/bespokoid/205efd91546ddb16b210678830...Sure those are good designs... But Lucen is not about "here's a nicer way to write loops," it's "you shouldn't have to touch the loop you already have."..... showing Lucen parallelize the ugly version is the point.
This code seems to be fairly pervasively AI-written (both structurally/syntacticly, with many additional AI verbosity tells in the documentation, and going by the commit history).
That's a problem for me, given what this library does: arbitrary transformations of Python code at load time. The fragility, security risk, and complexity added by basically writing a subdialect of Python with new semantics is something I'd like to see more human attention on and maturity of before I consider using it. There are times when AI smell doesn't concern me much when deciding whether or not to use someone's code. This isn't one of those times.
I think I’m a little skeptical.
#pragma is a real keyword in C but # is just a comment in python - why not use a pythonic @decorator?
Vendoring a library like this into an app seems like a lot more toil than just writing the native multiprocessing python code, or using something built for number crunching like arrayfire or numpy
Because (even if you could decorate for-loops, or instead decorated functions containing a loop and nothing else) the resulting behavior would be completely unlike the behavior of all other @decorators in Python. It would also be nearly impossible to implement this functionality as a decorator without introducing tons of fragility.
The preprocessor/import-hook hack that allows macro-like behavior in Python does arbitrary transforms at the text/code-load level, while decorators are evaluated at runtime and run regular Python code. Decorators can be composed/wrapped by other functions and decorators, whereas implementing Lucen-like functionality with a decorator would necessarily match on the name/symbol of the decorator and couldn't indirect "through" it.
Heck, even if you disallowed wrapping and matched on the decorator name during preprocessing (basically stripping it out and rewriting decorated code strings at load time), even identifying the correct decorator as a transformation target would require reimplementing a ton of Python's import/scope/name resolution behavior by hand at code-transformation time. If your decorator was called lucen.parallelize(), consider the difference between "import lucen; @lucen.parallelize", "from lucen import parallelize; @parallelize", "from lucen import parallelize as pl; parallelize = 123; @pl", and so on. You'd have to handle all of those cases, and many more (e.g. decorated functions defined inside other classes/functions) at the code-as-string or AST level. You couldn't run the decorator at runtime, because a) syntactic information is gone by then, and b) because then you'd need to unimport/reimport module code which had already been run with arbitrary global side effects.
More information on the methods available to get preprocessor-like functionality in Python:
Custom source encoding text transformers can be loaded via .pth files: https://pydong.org/posts/PythonsPreprocessor/
Import hooks can be configured at runtime, and run during subsequent import statements. They were originally designed to allow customized module discovery, but nothing stops you from using them like Lucen does: to intercept imports of local Python files and transform/replace the code imported at load time: https://peps.python.org/pep-0302/
You can @decorate a for-loop?
I wish there were easier ways to play with syntax in Python, but there we are.
Though I guess using @decorator semantics would make it look a lot like python-numba syntax
just wanted to make sure ... even if someone someday wishes to revert the decision of using lucen they will be okay with just removing the import hooks - and run everything as native -- and when they have time, they can clear the comments but till then they can work without thinking
Interesting to see the rust code is doing the actual parallelization, while execution. Will it work for complex loops, and what happens if the loops uses global context and different variables, outside of the loop?. May be it should go to python standard library.
This should be part of the Standard Library if so efficient.