Why Postgres Should be your Document Database Webinar Recording

5 Responses to Why Postgres Should be your Document Database Webinar Recording

1. 

   Peter says:

   March 25, 2017

   Speaker talks about scalability, proposes select nextval(tablename) to get next id safely. I’ve heard enough.

   • 

     Guy says:

     April 5, 2017

     What’s the problem with doing that? (I’m really asking for educational purposes)

     As far as I know, the database will make sure there are no conflicts in this case, no?

2. 

   Alejandro Villamarin says:

   March 28, 2017

   That was pretty educational, great topics covered. I have a question for Jim, though…when he talks about using pq for queuing emails…has he ever tried to do this along with Celery? Just wondering if this can substitute something like RabbitMQ or Redis, and why it should be better. I see some advantages in having a table with queued messages, but you might be shooting yourself in the foot trying to do something better than RabbitMQ?

   Thanks for a great talk!

   • 

     Jim Fulton says:

     March 28, 2017

     The problem with Celery (and SQS) is that the handoff isn’t
     transactional. This is a problem.

     I highlighted pq because there is at least a solution that has a
     transactional handoff.

     But you’re right, managing queues is hard, especially when dealing
     with various worker errors, and using something established is
     attractive.

     There was another solution, zc.async, that provided queues based on
     ZODB and it was very complex.

     I think this is mostly an unsolved (and sadly unrecognized problem).

     A way I solved this in the past for SQS was to use a very simple
     transactional database queue whose job was to very temporarily hold
     jobs to be submitted to SQS. Jobs were moved from the database-based
     queue to SQS after commit and, in the rare cases that that failed, the
     jobs were retried later using a clean-up process. The database-bases
     queue can be extremely simple because the workers only have one thing
     to do and failure is rare and virtually guaranteed to succeed
     eventually, generally on the first retry. I could easily imagine pq
     being used this way.

     It would be useful to have a transactional broker for celery.

     • 

       Alejandro says:

       April 6, 2017

       Good point, Jim.

       I’ll give pq a try, to see how it behaves dealing with asynch queue processing.

       Thanks!