Skip to content

Latest commit

 

History

History
98 lines (72 loc) · 3.85 KB

throwing_together_distributed_services_with_gevent.rst

File metadata and controls

98 lines (72 loc) · 3.85 KB
Raw

Throwing Together Distributed Services With Gevent (Ginkgo) by Jeff Lindsay from Twilio

Presenter: Jeff Lindsay (http://progrium.com/) (@progrium)

PyCon 2012 presentation page: https://us.pycon.org/2012/schedule/presentation/288/

Tutorial: https://github.com/progrium/ginkgotutorial

Slides: http://dl.dropbox.com/u/2096290/GinkgoPyCon.pdf

Video: http://pyvideo.org/video/642/throwing-together-distributed-services-with-geven

Track: V

Description

In this talk we learn how to throw together a distributed system using gevent and a simple framework called Ginkgo (formerly gservice). We'll go from nothing to a distributed messaging system ready for production deployment based on experiences building scalable, distributed systems at Twilio.

Abstract

As some have found, gevent is one of the best kept secrets of Python. It gives you fast, evented network programming without messes of callbacks, code that is more Pythonic, and lets you use most regular Python networking libraries and protocol implementations. Now, let's build on this.

In this talk we learn how to throw together distributed services using gevent and a simple framework called Ginkgo (formerly gservice). We'll go from nothing to a distributed messaging system based on experiences building scalable, distributed systems at Twilio.

This talk will be full of code, live coding, and real production applications with guest appearances by other fun technologies like ZeroMQ, WebSocket, and Doozer.

Jeff works at Twilio. Twilio uses a service oriented architecture. Twilio as a high level service is made up of many subservices - sms, voice, client. Each is made up of other services. Mostly written in tornado and gevent. Going to use a framework called Ginkgo.

  • services are nested modules that can start, stop and reload
  • going to build a scalable gateway around a self-organizing messaging system.
  • first we'll build a simple number client
  • next a pubsub service
  • then combine both into a gateway service
  • and finally make it all distributed

Number Server

  • a basic gevent StreamServer
  • for every connection generate a random number, sleep for 60 seconds
  • configuration is a python file
  • ginkgo has start / stop services
  • start the number server in the background

Number Client

  • spawns a greenlet relative to your service
  • makes services self-contained
  • gets the numbers from the server and puts it in a queue

PUBSub

  • uses httpstreamer
  • subscription wraps a queue
  • posts are publishes, gets are subscribes

MessageHub

  • hub is now responsible for managing subscriptions

If you're in a loop that doesn't use IO, run gevent.sleep(0) to make sure it yields.

Code for the example is up here: https://github.com/progrium/ginkgotutorial