A Python 3 microservice library / framework using asyncio (async / await) with HTTP, websockets, RabbitMQ / AMQP and AWS SNS+SQS built-in support for event based messaging and intra-service communication.
Tomodachi is a tiny framework designed to build fast microservices listening on HTTP or communicating over event driven message buses like RabbitMQ, AMQP, AWS (Amazon Web Services) SNS+SQS, etc. It's designed to be extendable to make use of any type of transport layer available.
Tomodachi [ει] means friends β a suitable name for microservices working together. π» π¬ π π« π»
tomodachi
is invoked via command line interface.
Usage: tomodachi <subcommand> [options] [args] Options: -h, --help show this help message and exit -v, --version print tomodachi version --dependency-versions print versions of dependencies Available subcommands: run <service ...> [-c <config-file ...>] [--production] -c, --config <files> use json configuration files -l, --log <level> specify log level --production disable restart on file changes
- Installation
- Getting started / example services
- Running microservices in Docker
- Defining endpoints
- Requirements
- Questions
- Contributions
tomodachi
is still a highly experimental project with an unregular release
schedule.
Preferrably installation should be done via pip
to get the cli alias set
up automatically. Locally it is recommended to install tomodachi
into a
virtualenv to avoid random packages into your base site-packages.
local ~$ pip install tomodachi
Start off with import tomodachi
and add a class decorated with
@tomodachi.service
and/or extended from the tomodachi.Service
class.
Name your service class and then just add functions and triggers for how to
invoke them, either by HTTP requests, event messages or by timestamps /
intervals.
Code for a simple service which would service data over HTTP.
import tomodachi
@tomodachi.service
class Service(tomodachi.Service):
name = 'example'
# Request paths are specified as regex for full flexibility
@tomodachi.http('GET', r'/resource/(?P<id>[^/]+?)/?')
async def resource(self, request, id):
# Returning a string value normally means 200 OK
return 'id = {}'.format(id)
@tomodachi.http('GET', r'/health')
async def health_check(self, request):
# Return can also be a tuple, dict or even an aiohttp.web.Response
# object for more complex responses - for example if you need to
# send byte data, set your own status code or define own headers
return {
'body': 'Healthy',
'status': 200
}
# Specify custom 404 catch-all response
@tomodachi.http_error(status_code=404)
async def error_404(self, request):
return 'error 404'
Example of a service that would invoke a function when messages are published on a topic exchange.
import tomodachi
@tomodachi.service
class Service(tomodachi.Service):
name = 'example'
# A route / topic on which the service will subscribe to via AMQP (or AWS SNS/SQS)
@tomodachi.amqp('example.topic')
async def example_topic_func(self, message):
# Received message, sending same message as response on another route / topic
await tomodachi.amqp_publish(self, message, routing_key='example.response')
There are other examples available with examples of how to use services with self-invoking methods called on a specified interval or at specific times / days. Inter-communication between different services may be established using a pub-sub type with messages over AMQP or AWS SNS+SQS which is natively supported.
See a more comprehensive example involving multiple services publishing and subcribing on topics using AWS SNS+SQS in the pubsub-examples folder.
# cli alias is set up if installed via pip
local ~/code/service$ tomodachi run service.py
# example if cloned from repo
local ~/code/tomodachi$ python tomodachi.py run example/http_simple_service.py
Defaults to output information on stdout.
local ~/code/service$ tomodachi run service.py
tomodachi/X.X.XX
October 02, 2017 - 13:38:00,481516
Quit services with <ctrl+c>.
2017-10-02 13:38:01,234 (services.service): Initializing service "example" [id: <uuid>]
2017-10-02 13:38:01,248 (transport.http): Listening [http] on http://127.0.0.1:9700/
2017-10-02 13:38:01,248 (services.service): Started service "example" [id: <uuid>]
HTTP service acts like a normal web server.
local ~$ curl -v "http://127.0.0.1:9700/resource/1234"
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Server: tomodachi
< Content-Length: 9
< Date: Mon, 02 Oct 2017 13:38:02 GMT
id = 1234
Great ways to run microservices are either to run them in Docker or running them serverless.
Here's an example of getting a tomodachi service up and running in Docker in no-time. The
base-image (kalaspuff/python-nginx-proxy
) also sets up nginx
and proxies requests from
port 80 to the service backend on 8080.
We're building a container using just two small files, the Dockerfile
and the actual code
for the microservice, service.py
.
Dockerfile
FROM kalaspuff/python-nginx-proxy:1.2.1
WORKDIR /
RUN apt-get -y update \
&& apt-get install -y build-essential=12.3 \
&& pip install tomodachi \
&& apt-get purge -y --auto-remove build-essential \
&& apt-get clean autoclean \
&& apt-get autoremove -y \
&& rm -rf /var/lib/{apt,dpkg,cache,log}/
RUN mkdir /app
WORKDIR /app
ADD service.py .
CMD tomodachi run service.py --production
service.py
import tomodachi
@tomodachi.service
class Service(tomodachi.Service):
name = 'example'
options = {
'http': {
'port': 8080
}
}
@tomodachi.http('GET', r'/')
async def index_endpoint(self, request):
return 'friends forever!'
Building and running the container, forwarding host's port 31337 to port 80.
local ~/code/service$ docker build . -t tomodachi-microservice
local ~/code/service$ docker run -ti -p 31337:80 tomodachi-microservice
2017-10-02 13:38:01,234 (services.service): Initializing service "example" [id: <uuid>]
2017-10-02 13:38:01,248 (transport.http): Listening [http] on http://127.0.0.1:8080/
2017-10-02 13:38:01,248 (services.service): Started service "example" [id: <uuid>]
Making requests to the running container.
local ~$ curl http://127.0.0.1:31337/
friends forever!
Nothing more nothing less. It's actually as easy as that.
There are several built-in ways to invoke your microservice methods in which the most common ones are either directly via HTTP or via event based messaging (for example AMQP or AWS SNS+SQS). Here's a list of the currently available built-ins you may use to decorate your service functions. Here's a short run-down of the available decorators.
@tomodachi.http(method, url, ignore_logging=[200])
- Sets up an HTTP endpoint for the specified
method
(GET
,PUT
,POST
,DELETE
) on the regexpurl
. Optionally specifyignore_logging
as a dict or tuple containing the status codes you do not wish to log the access of. Can also be set toTrue
to ignore everything except status code 500. @tomodachi.http_static(path, url)
- Sets up an HTTP endpoint for static content available as
GET
/HEAD
from thepath
on disk on the base regexpurl
. @tomodachi.websocket(url)
- Sets up a websocket endpoint on the regexp
url
. The invoked function is called upon websocket connection and should return a two value tuple containing callables for a function receiving frames (first callable) and a function called on websocket close (second callable). @tomodachi.http_error(status_code)
- A function which will be called if the HTTP request would result in a 4XX
status_code
. You may use this for example to set up a custom handler on "404 Not Found" or "403 Forbidden" responses.
@tomodachi.aws_sns_sqs(topic, competing=None, queue_name=None, **kwargs)
This would set up an AWS SQS queue, subscribing to messages on the AWS SNS topic
topic
, whereafter it will start consuming messages from the queue.The
competing
value is used when the same queue name should be used for several services of the same type and thus "compete" for who should consume the message.Unless
queue_name
is specified an auto generated queue name will be used. Additional prefixes to bothtopic
andqueue_name
can be assigned by setting theoptions.aws_sns_sqs.topic_prefix
andoptions.aws_sns_sqs.queue_name_prefix
dict values.Depending on the service
message_protocol
used, parts of the enveloped data would be distribbuted to different keyword arguments of the decorated function. It's usually safe to just usedata
as an argument. You can also specify a specificmessage_protocol
value as a keyword argument to the decorator for specifying a specific enveloping method to use instead of the global one set for the service.If you're utilizing
from tomodachi.protocol import ProtobufBase
and usingProtobufBase
as the specified servicemessage_protocol
you may also pass a keyword argumentproto_class
into the decorator, describing the protobuf (Protocol Buffers) generated Python class to use for decoding incoming messages.
@tomodachi.amqp(routing_key, exchange_name='amq.topic', competing=None, queue_name=None, **kwargs)
Sets up the method to be called whenever a AMQP / RabbitMQ message is received for the specified
routing_key
. By default the'amq.topic'
topic exchange would be used, it may also be overridden by setting theoptions.amqp.exchange_name
dict value for the service class.The
competing
value is used when the same queue name should be used for several services of the same type and thus "compete" for who should consume the message.Unless
queue_name
is specified an auto generated queue name will be used. Additional prefixes to bothrouting_key
andqueue_name
can be assigned by setting theoptions.amqp.routing_key_prefix
andoptions.amqp.queue_name_prefix
dict values.Depending on the service
message_protocol
used, parts of the enveloped data would be distribbuted to different keyword arguments of the decorated function. It's usually safe to just usedata
as an argument. You can also specify a specificmessage_protocol
value as a keyword argument to the decorator for specifying a specific enveloping method to use instead of the global one set for the service.If you're utilizing
from tomodachi.protocol import ProtobufBase
and usingProtobufBase
as the specified servicemessage_protocol
you may also pass a keyword argumentproto_class
into the decorator, describing the protobuf (Protocol Buffers) generated Python class to use for decoding incoming messages.
@tomodachi.schedule(interval=None, timestamp=None, timezone=None, immediately=False)
A scheduled function invoked on either a specified
interval
(you may use the popular cron notation as a str for fine-grained interval or specify an integer value of seconds) or a specifictimestamp
. Thetimezone
will default to your local time unless explicitly stated.When using an integer
interval
you may also specify wether the function should be calledimmediately
on service start or wait the fullinterval
seconds before its first invokation.@tomodachi.heartbeat
- A function which will be invoked every second.
@tomodachi.minutely
,@tomodachi.hourly
,@tomodachi.daily
,@tomodachi.monthly
- A scheduled function which will be invoked once every minute / hour / day / month.
You may also extend the functionality by building your own transports for your endpoints. The invokers themselves should extend the class tomodachi.invoker.Invoker
.
Invoker functions can of course be decorated using custom functionality. For ease of use you can then in turn decorate your decorator with the the built-in @tomodachi.decorator
to ease development.
If the decorator would return anything else than True
or None
(or not specifying any return statement) the invoked function will not be called and instead the returned value will be used, for example as an HTTP response.
import tomodachi
@tomodachi.decorator
async def require_csrf(instance, request):
token = request.headers.get("X-CSRF-Token")
if not token or token != request.cookies.get('csrftoken'):
return {
'body': 'Invalid CSRF token',
'status': 403
}
@tomodachi.service
class Service(tomodachi.Service):
name = 'example'
@tomodachi.http('POST', r'/create')
@require_csrf
async def create_data(self, request):
# Do magic here!
return 'OK'
Offered under the MIT license
The latest developer version of tomodachi
is available at the GitHub repo https://github.com/kalaspuff/tomodachi
- What is the best way to run a
tomodachi
service? There is no way to tell you how to orchestrate your infrastructure. Some people may run it containerized in a Docker environment, deployed via Terraform / Nomad / Kubernetes and some may run several services on the same environment, on the same machine. There may be best practices but theres no way telling you how to orchestrate your application environment.
Personally I would currently go for a Dockerized environment with nginx proxy in front of the service to handle all the weirdness of the web, TLS, black magic and improved upgrades for WebSockets. Take a look at my kalaspuff/docker-python-nginx-proxy base-image to get your code up and running within minutes.
- Are there any more example services?
- There are a few examples in the examples folder, including using
tomodachi
in an example Docker environment with or without docker-compose, there are examples to publish events/messages to an AWS SNS topic and subscribe to an AWS SQS queue. There's also a similar example of how to work with pub-sub for RabbitMQ via AMQP transport protocol. - Why should I use this?
tomodachi
is a perfect place to start when experimenting with your architecture or trying out a concept for a new service. It may not have all the features you desire and it may never do, but I believe it's great for bootstrapping microservices in async Python.- I have some great additions!
- Sweet! Please send me a PR with your ideas. Get started at the short contribution guide.
- Should I run this in production?
Yes? No? There are some projects that already have live versions in production. The library is provided as is with an unregular release schedule. It's all still highly experimental and it depends on other experimental projects, so you have to be in charge here and decide for yourself. Let me know if you do however!
Another good idea is to drop in Sentry or other exception debugging solutions, for if your invoked functions would raise unhandled exceptions.
- Who built this and why?
- My name is Carl Oscar Aaro [@kalaspuff] and I'm a coder from Sweden. I simply wanted to learn more about asyncio and needed a constructive off-work project to experiment with β and here we are. Nowadays I use
tomodachi
as a base for many smaller projects where I just want to be able to focus on the application itself, while still having the power of building distributed systems. π
Please help out to add features that you deem are missing and/or fix bugs in the repo.
To add a PR, for the repository, commit your changes to your own clone and make a PR on GitHub for your clone against master branch.
Read more in the contribution guide.