A clean, lightweight alternative to Python?


jkp :

One (a long time ago) I wrote a web spider and I multithreaded the thread to allow concurrent requests to happen at the same time. That was my Python youth, before I learned about the GIL and its associated troubles for multithreaded code (IE, most of the time, these things end up being serialized!)...

I would like to redo this code to make it more robust and perform better. There are basically two ways to do this: I could use the new multiprocessing module in 2.6+ , or I could use some kind of reactor/event based model. I'd rather do it later because it's much simpler and less error-prone.

So the question is about which framework is best for my needs. Here is the list of options I know so far:

  • Twisted : The grandfather of the Python reactor framework: It looks complicated, but it's a bit bloated. Steep learning curve for small tasks.
  • Eventlet : From the guys at lindenlab . A Greenlet-based framework for such tasks. I took a look at the code, but it doesn't look pretty: it's not Pep8 compliant, with prints scattered around (why would people do this in a framework!?), the API seems a bit inconsistent.
  • PyEv : immature, although it's based on libevent, nobody seems to be using it, so it has a solid backend.
  • asyncore : from stdlib: über low level, seems like a lot of work involved just to get things off the ground.
  • Tornado : Although this is a server-oriented product designed to serve dynamic websites, it does have an asynchronous HTTP client and a simple ioloop . Seems to get the job done, but not as intended. [EDIT: Unfortunately it doesn't work on Windows, which counts as it for me - it's my requirement to support this lame platform]

Is there anything I'm missing? Of course, there must be a library that fits in the sweet spot of simplifying async networking libraries!

[Edit: Many thanks to intgr for pointing to this page . If you scroll to the bottom, you will see a very nice list of projects designed to solve this task in one way or another. In fact, things have indeed changed since Twisted was born: people now seem to prefer coroutine -based solutions over traditional reactor/callback oriented solutions. The benefit of this approach is more straightforward code: I did find it in the past, especially when using boost.asio. In C++, callback-based code can lead to hard-to-follow designs and is relatively obscure to the untrained. Using coroutines allows you to write code that looks at least more synchronous. I guess my task now is to find out which of the many libraries I like and give it a try! Glad I'm asking now...]

[EDIT: May be of interest to anyone following or stumbled upon this question or the subject in any sense: I found a really good article on the current state of the tools available for the job ]

Integer:

I like the Concurrent Python module which relies on Lightweight Threaded Stackless Python Microthreads or Greenlets. All blocking network I/O can be transparently async through a libeventloop , so it should be about as efficient as a true async server.

I guess this way is similar to Eventlet.

The downside is that its API is completely different from Python's sockets/ threadingmodules ; you'll need to rewrite parts of your application (or write compatibility padding)

Edit: There also appears to be cogen , which is similar, but uses Python 2.5's enhanced generators for its coroutines instead of Greenlets. This makes it more portable than concurrency and other alternatives. Network I/O is done directly via epoll/kqueue/iocp.

Related


A clean, lightweight alternative to Python?

jkp : One (a long time ago) I wrote a web spider and I multithreaded the thread to allow concurrent requests to happen at the same time. That was my Python youth, before I learned about the GIL and its associated troubles for multithreaded code (IE, most of th

A clean, lightweight alternative to Python?

jkp : One (a long time ago) I wrote a web spider and I multithreaded the thread to allow concurrent requests to happen at the same time. That was my Python youth, before I learned about the GIL and its associated troubles for multithreaded code (IE, most of th

Lightweight alternative to Hibernate?

Jared: I have a user Java program that wants to store data in a lightweight database such as Derby or Sqlite. I want to use a data abstraction layer in my program. Hibernate seems to require a lot of configuration and is overkill for what I need. What is a lig

Lightweight alternative dock

Alex: Generally , Jetty is known as a lightweight alternative when it comes to servlet containers like Tomcat and App Servers like Glassfish . I want to run a RESTful service on CloudFoundry. use jetty java -jar target/dependency/jetty-runner.jar target/*.war

Lightweight GNU readline alternative

punekr12 : I am looking for a GNU readline alternative. It has many features but only a few of them worked for me as described below - I am developing an interactive command prompt application (displays the prompt and accepts the next user command to run). I w

Lightweight multilingual alternative to JMX?

ROM1: I have a software real-time application project that requires extensive monitoring. JMX seems to be a good fit for the task, it's just that the application project is based on c++. Is there a lightweight alternative to JMX (with a c/c++ support library)

Lightweight alternative dock

Alex: Generally , Jetty is known as a lightweight alternative when it comes to servlet containers like Tomcat and App Servers like Glassfish . I want to run a RESTful service on CloudFoundry. use jetty java -jar target/dependency/jetty-runner.jar target/*.war

Lightweight alternative to Hibernate?

Jared: I have a user Java program that wants to store data in a lightweight database such as Derby or Sqlite. I want to use a data abstraction layer in my program. Hibernate seems to require a lot of configuration and is overkill for what I need. What is a lig

Lightweight GNU readline alternative

punekr12 I am looking for a GNU readline alternative. It has many features but only a few of them worked for me as described below - I am developing an interactive command prompt application (displays the prompt and accepts the next user command to run). I wan

Lightweight GNU readline alternative

punekr12 : I am looking for a GNU readline alternative. It has many features but only a few of them worked for me as described below - I am developing an interactive command prompt application (displays the prompt and accepts the next user command to run). I w

Lightweight multilingual alternative to JMX?

ROM1: I have a software real-time application project that requires extensive monitoring. JMX seems to be a good fit for the task, it's just that the application project is based on c++. Is there a lightweight alternative to JMX (with a c/c++ support library)

Lightweight zxcvbn alternative?

Greg Blass Wondering if there are any lightweight zxcvbn alternatives out there. I don't need my password strength meter to be incredible - rather, I need to reduce the size of my assets. Can someone say 80% of the content? Maybe not the whole dictionary in th

Lightweight alternative dock

Alex Generally , Jetty is known as a lightweight alternative when it comes to servlet containers like Tomcat and App Servers like Glassfish . I want to run a RESTful service on CloudFoundry. use jetty java -jar target/dependency/jetty-runner.jar target/*.war

Lightweight alternative to Hibernate?

Jared: I have a user Java program that wants to store data in a lightweight database such as Derby or Sqlite. I want to use a data abstraction layer in my program. Hibernate seems to require a lot of configuration and is overkill for what I need. What is a lig

Lightweight alternative dock

Alex: Generally , Jetty is known as a lightweight alternative when it comes to servlet containers like Tomcat and App Servers like Glassfish . I want to run a RESTful service on CloudFoundry. use jetty java -jar target/dependency/jetty-runner.jar target/*.war

Lightweight GNU readline alternative

punekr12 : I am looking for a GNU readline alternative. It has many features but only a few of them worked for me as described below - I am developing an interactive command prompt application (displays the prompt and accepts the next user command to run). I w

Lightweight multilingual alternative to JMX?

ROM1: I have a software real-time application project that requires extensive monitoring. JMX seems to be a good fit for the task, it's just that the application project is based on c++. Is there a lightweight alternative to JMX (with a c/c++ support library)

Lightweight alternative to fork() in POSIX C?

Kyle From the man pages I'm reading, it seems that popen, system, etc. tend to call fork(). In turn, fork() copies the entire memory state of the process. This does seem like a lot of work, especially in many cases where the child calling fork() barely has any

Is there a (lightweight) alternative to `rxvt-unicode`?

Buffalo I am currently rxvt-unicodeusing as a terminal emulator. Since I also like the configurability of GNOME and KDE's terminal emulators, I'm wondering if there is some sort of alternative that can replace GNOME or KDE Terminal, but rxvt-unicodewith more f

Lightweight alternative to fork() in POSIX C?

Kyle From the man pages I'm reading, it seems that popen, system, etc. tend to call fork(). In turn, fork() copies the entire memory state of the process. This does seem like a lot of work, especially in many cases where the child calling fork() barely has any

Is there a (lightweight) alternative to `rxvt-unicode`?

Buffalo I am currently rxvt-unicodeusing as a terminal emulator. Since I also like the configurability of GNOME and KDE's terminal emulators, I'm wondering if there is some sort of alternative that can replace GNOME or KDE Terminal, but rxvt-unicodewith more f

Lightweight alternative to fork() in POSIX C?

Kyle From the man pages I'm reading, it seems that popen, system, etc. tend to call fork(). In turn, fork() copies the entire memory state of the process. This does seem like a lot of work, especially in many cases where the child calling fork() barely has any

Lightweight alternative to fork() in POSIX C?

Kyle From the man pages I'm reading, it seems that popen, system, etc. tend to call fork(). In turn, fork() copies the entire memory state of the process. This does seem like a lot of work, especially in many cases where the child calling fork() barely has any

Is there a (lightweight) alternative to `rxvt-unicode`?

Buffalo I am currently rxvt-unicodeusing as a terminal emulator. Since I also like the configurability of GNOME and KDE's terminal emulators, I'm wondering if there is some sort of alternative that would replace GNOME or KDE Terminal, but rxvt-unicodewith more

Alternative to quick builds - clean?

Philip So, it used to be common to clean builds from the command line swift build --cleanby deleting .builddirectories . You can also --clean=distget rid of the package via . I just upgraded to Swift 4 and these are apparently gone. Has it been rolled into sub

Alternative to quick builds - clean?

Philip So, it used to be common to clean builds from the command line swift build --cleanby deleting .builddirectories . You can also --clean=distget rid of the package via . I just upgraded to Swift 4 and these are apparently gone. Has it been rolled into sub

Alternative to quick builds - clean?

Philip So, it used to be common to clean builds from the command line swift build --cleanby deleting .builddirectories . You can also --clean=distget rid of the package via . I just upgraded to Swift 4 and these are apparently gone. Has it been rolled into sub

Best alternative to String lightweight implementation in Java

and: 我的应用程序是带有密集字符串处理的多线程。我们正在经历过多的内存消耗,并且性能分析表明这是由于String数据引起的。我认为使用某种flyweight模式实现甚至是缓存将极大地受益于内存消耗(我可以肯定Strings通常是重复的,尽管我在这方面没有任何硬数据)。 我看过Java常量池和String.intern,但似乎可以引发一些PermGen问题。 在Java中实现应用程序范围的多线程字符串池的最佳替代方法是什么? 编辑:另请参阅我以前的相关问题:Java如何在引擎盖下实现字符串的flyweight模式

Best alternative to String lightweight implementation in Java

and: My application is multithreaded with intensive string processing. We are experiencing excessive memory consumption and profiling shows that it is due to String data. I think implementing or even caching with some kind of flyweight pattern would greatly be