Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you're working on machine learning the most economic choice is Python.

But weiting a processing pipeline with Python is frustrating if you have worked with C# concurrency.

I figured the best option is Celery and you cannot do it without an external broker. Celery is a mess. I really hate it.





Agree. I think it's improved a bit but Celery is frustrating as the defacto job/queue solution. A lot of the defaults make it unreliable (it can lose jobs if workers crash or don't shutdown cleanly)

I'm hoping the existence of free-threading will push for more first-class concurrency primitives. Concurrent Futures is nice until you need a concurrent-safe data structure besides a queue


Agree that celery is a mess and it doesn't work well with async (Asyncio) python. I think version 6 maybe will support it sometime.

I also had a lot of problem due to async primitives with sqlalchemy - there's some tricky stuff with asyncio.gather vs TaskGroup and how sqlalchemy session works with it to be able to compose code easily.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: