Get the code from Github
Clay is a framework for building RESTful backend services using best practices. It’s a wrapper around Flask (http://flask.pocoo.org) along with several convenient, standardized, performance enhanced ways of performing common tasks like sending email and connecting to a database. Clay is available under the MIT License.
If you’re developing a new service, you’ve come to the right place. Let’s start with the obligatory Hello World. You’ll notice that this is very similar to Flask’s Hello World example.
link:docs/helloworld.py[role=include]A few noteworthy things here that begin our list of best practices…
-
Every module starts with
from __future__ import absolute_import, this is to prevent large projects from ending up with cyclical imports. -
Along those same lines, wildcard imports like
from foo import *should never be used. Always explicitly list the things you’re importing from a module. -
fromimports should be avoided when practical. In the middle of a thousand-line file, it might not be obvious thatmail.send(...)is reallyclay.mail.send(...)and not an instance of a local object. -
Keep reasonably close to PEP8 recommendations. Four space tabs, two newlines between top-level definitions in a module, etc. If you like keeping things to 80 columns, that’s fine, but if a line needs to be longer, nobody’s going to complain.
-
Explicitly list the methods that a route responds to. We know that
app.route()defaults to GET methods only, but this is not guaranteed to be the same in future versions of Flask/Clay.
Every Clay application needs a config. Clay configs are JSON files. The clay framework itself looks for a few config variables (TODO: see below). By default, we look for files listed in $CLAY_CONFIG delimited by colons : relative to the directory the development server runs from. Here’s a simple example.
link:docs/simple-clay.conf[role=include]This is the minimum configuration necessary to run the clay devserver. The first flask init section is the module name passed to the Flask app constructor. This is only used for uniquely identifying this application internally and has no bearing on anything important. It should follow the naming rules for Python modules. Here we use helloworldapp just to differentiate it from the name of our view module.
The debug server section provides a host and port for the clay devserver to listen on. No magic here, just an IP and port. Use "0.0.0.0" for the host to listen on all interfaces.
views is a list of modules to be loaded when the server starts up. This allows you to run multiple view modules simultaneously without having to manage a file with a list of routes or prefixes.
Now that we’ve got a view module and a config, we can run the clay devserver…
export CLAY_CONFIG=./simple-clay.conf
clay-devserverCLAY_CONFIG is a colon delimited list of config files to be loaded, in order of precedence. For example, if CLAY_CONFIG=./common.conf:./janky.conf, the configuration dictionary from common.conf is loaded first, then update()'d with the dictionary from janky.conf.
Clay exposes itself as a standard WSGI application in the module
clay.wsgi. The CLAY_CONFIG environment variable needs to be set.
All view modules listed in the config will be imported at startup.
export CLAY_CONFIG=./simple-clay.conf
gunicorn clay.wsgi:applicationThe clay.config module provides a simple API for accessing
configuration information. Internally, all Clay modules use this
module to configure themselves.
Upon import, clay.config attempts to load it’s configuration from
files listed in the CLAY_CONFIG environment variable, delimited by
colons. Each file’s name is examined for a known file extension and
parsed by the appropriate deserializer. Currently json is supported
using the standard library and yaml is supported if PyYAML is
installed. As each file is parsed, it is applied to the global config
with dict.update().
This module registers itself as a handler for SIGHUP and will attempt to reload it’s configuration upon receiving that signal. The configuration may also be reloaded on demand by calling config.load().
Several methods are exposed at the top level of the clay.config module and are intended to provide the config’s public API.
key is a period . delimited string that is recursively searched for a configuration option with that name. If this lookup fails at any level of the config hierarchy, the value of default is returned. For example, you can expect the following behavior for the given config.
{
"users": {
"admins": ["synack", "bigo"]
}
}>>> from clay import config
>>> config.get('users.admins')
[u"synack", u"bigo"]
>>> config.get('users.players')
None
>>> config.get('dogs')
None
>>> config.get('dogs', default=True)
TrueReturns a pre-configured logging.Logger instance identified by the given name. Depending on the framework’s environment, this logger may format and emit messages in different ways. In development, messages will be routed to the console whereas in production they might be routed to an aggregate endpoint or archive.
>>> from clay.config import config
>>> log = config.get_logger('myservice')
>>> log.debug('Now we know what\'s happening!')
myservice DEBUG Now we know what's happening!DEPRECATED, will be removed in a future release. Use
clay.config.get('debug.enabled', False) instead.
Similar to clay.config.get, feature_flag is specific to things with boolean values that enable or disable functionality within your service. This method returns True if the given feature is enabled, False otherwise.
{
"features": {
"new_shiny_bits": {
"enabled": true
},
"new_scary_thing": {
"enabled": true,
"percent": 10.0
}
}
}In the example above, feature_flag('new_shiny_bits') would return True and feature_flag('new_scary_thing') will only return True 10% of the time. The percent option is useful for A/B testing new features or slowly rolling out a feature for a subset of requests to gauge performance.
Sends an email to the given address using the server/credentials specified in the config under smtp.
Additional SMTP headers may be set as keyword arguments, the values of which are expected to be either a subclass of basestring or an iterable of basestrings.
By default the From address is populated by the smtp.from config option. This may be overridden by passing a From kwarg.
{
"smtp": {
"host": "smtp.example.com",
"port": 25,
"username": "myname",
"password": "superseekrit",
"from": "[email protected]"
}
}from clay import mail
# Simple example
mail.sendmail('[email protected]', 'Not spam I promise!', 'A simple example')
# Complex example
mail.sendmail('[email protected]', 'Definitely not spam', CC=[
'[email protected]',
'[email protected]'
], BCC='[email protected]', From='[email protected]', subtype='html',
message='<marquee><blink>YOU OBVIOUSLY LOVE OWLS</blink></marquee>')Performs an HTTP request and returns a Response object (which is just a namedtuple) with status, headers, and data attributes. This module is just a wrapper around urllib2 so any installed openers or redirect handlers will be used. See the urllib2 docs for more information (http://docs.python.org/2/library/urllib2.html).
from clay import http
response = http.request('GET', 'http://httpbin.org/ip')
if response.status != 200:
print 'Something bad happened: ', response.dataDecorator that adds a Cache-Control header to the response of a view
function. Each keyword argument to this decorator is appended to the
Cache-Control value, delimited by a comma. The key of each argument
has any underscore _ characters replaced with dash -.
from clay import app, http
@app.route('/hello_cache', methods=['GET']
@http.cache_control(max_age=3600, public=True)
def hello_cache():
return 'Hi!'HTTP/1.1 200 OK Cache-Control: max-age=3600, public
A lightweight context manager around DB-API (PEP 249) to provide access to SQL databases with read/write splitting. sqlite3, psycopg2, and MySQLdb modules are supported, although SQL syntax may vary.
In the configuration, lists of read and write servers are configured. You must configure at least one of each, even if both sections only point to a single server. If multiple servers are specified, a random one will be chosen every time the context manager is entered.
Connections to the database are opened upon entering the context manager and closed upon exiting. There is no support for connection pooling.
These are instances of a context manager with enter and exit functions that open a new database connection upon enter and close that connection upon exit.
from flask import request
from clay import app, database
@app.route('/user/create', methods=['POST'])
def user_create():
with database.write as db:
cur = db.cursor()
cur.execute('INSERT INTO users(email) VALUES(%s)', request.form['email'])
cur.close()
db.commit()
@app.route('/user/<int:userid>', methods=['GET'])
def user_show(userid):
with database.read as db:
cur = db.cursor()
cur.execute('SELECT email FROM users WHERE id=?', userid)
result = cur.fetchone()
cur.close()
return result- Config example
{
"database": {
"module": "psycopg2",
"read": [
{
"host": "readslave1.example.com",
"port": 5432,
"user": "readuser",
"password": "1234"
},
{
"host": "readslave2.example.com",
"port": 5432,
"user": "readuser",
"password": "1234"
}
],
"write": [
{
"host": "writemaster1.example.com",
"port": 5432,
"user": "writeuser",
"password": "4321"
}
]
}
}This module is not meant to be imported by an application, rather it
can be added to the views list in your app’s configuration to provide
a GET /_docs endpoint that parses the docstrings of all registered
routes and returns a JSON representation of your API’s documentation.
In order to provide enough metadata to generate documentation from a route, a machine readable format must be defined for writing docstrings. This module uses a format loosely based on Sphinx httpdomain (http://pythonhosted.org/sphinxcontrib-httpdomain/).
The first non-whitespace line of the docstring is assumed to be a "short doc" or "summary" of the endpoint’s behavior. This SHOULD be a human readable description of the endpoint less than 72 characters long. Lines following the short doc are a longer human readable description of the endpoint’s behavior.
Any line beginning with a : character is split on : and assumed to
expand to a tuple of (directive, key, value).
Within the value of a directive, if an opening { and closing }
brace are detected, the substring within the braces is parsed as a
JSON hash with additional metadata about this parameter. Currently
this is only used for specifying which parameters are required.
The :rtype: directive is special, it does not require a key but
references a dict object by import name. This dict defines the fields
of a JSON response body in a format defined in JSON-Schema format
utilized by swagger. (http://json-schema.org/)
(https://github.com/wordnik/swagger-core/wiki/datatypes)
@app.route('/things/search', methods=['GET'])
def hello():
'''
Search for things
This is an example of a search endpoint that takes query string
parameters to limit the scope of the search.
:query keywords: A whitespace delimited list of keywords to be included in the search. The implicit boolean AND is applied to all terms. {"required": true}
:query limit: The number of results to return, if not specified 100 is assumed
:status 200: Search completed, results are included in the response
:status 400: Required parameter keywords was not specified or limit was not an integer
:status 404: Search completed, but no results matched
:rtype: things.protocol.SearchResults
'''
# TODO: implement searching things
pass
SearchResults = {
"id": "SearchResults",
"required": ["count", "results"],
"properties": {
"count": {
"type": "integer"
},
"results": {
"type": "array",
"items": {
"$ref": "SearchResult"
}
}
}
}
SearchResult = {
"id": "SearchResult",
"required": ["name", "rank", "link"],
"properties": {
"name": {
"type": "string"
},
"rank": {
"type": "integer"
},
"link": {
"type": "string"
}
}
}Defines a query string argument.
:query <argname>: <description> Example: GET /foo?bar=baz :query bar: You might specify something like baz
Defines a POST application/x-www-form-urlencoded field
:form <argname>: <description> Example: POST /foo Content-type: application/x-www-form-urlencoded bar=baz :form bar: You might specify something like baz
Defines a field of a structured HTTP request body. The semantics of the body structure are not defined. In most cases, this will provide a reference to a JSON hash key.
:body <argname>: <description>
Example:
POST /foo
Content-type: application/json
{"bar": "baz"}
:body bar: You might specify something like baz
Defines a URL path part in reference to the view’s route.
:path <argname>: <description>
Example:
@app.route('/foo/<bar>', methods=['GET'])
def foo(bar):
pass
GET /foo/baz
:path bar: You might specify something like baz
:query <argname>: <description>
Example:
:query foo: Does foo like things {"required": true}
Sometimes when unittesting, it may make sense to test various configuration settings. Using a base unittest like so:
class CerebroTestCase(unittest.TestCase):
def __call__(self):
#Do pre-test stuff
self._pre_setup()
unittest.TestCase.__call__(self, result)
#Do post-test stuff
self._post_teardown()
def _pre_setup(self):
pass
def _post_teardown(self):
passSince clay’s configuration is stored as a dictionary, the trick is to use mock’s patch dictionary to replace the read-only dictionary with a malleable one:
class CerebroTestCase(unittest.TestCase):
def __call__(self):
# Copy configuration dictionary
self.config_copy = clay.config.CONFIG.config.copy()
# Do pre-test stuff
self._pre_setup()
# Patch config dictionary so tests can customize in setUp but also
# in the test function itself.
with mock.patch.object(clay.config.CONFIG, "config", self.config_copy):
unittest.TestCase.__call__(self, result)
# Do post-test stuff
self._post_teardown()
def _pre_setup(self):
pass
def _post_teardown(self):
passFinally, you can specify an attribute on a test with a dictionary of clay.config keys
and values to be applied to each test (in the example below, it’s test_config), as
well as a utility function so that the configuration can be manipulated for a single test:
class CerebroTestCase(unittest.TestCase):
...
def _pre_setup(self):
if hasattr(self, 'test_config'):
# Support ability for tests to have their own test settings
for attr, value in self.test_config.items():
self.set_config(attr, value)
def set_config(self, key, value):
"""Set a clay configuration option."""
val = self.config_copy
keys = key.split('.')
for k in keys[:-1]:
val.setdefault(k, {})
val = val[k]
val[keys[-1]] = valueAn example test using the above patterns:
class TestToast(CerebroTestCase):
# These values will apply to all tests in this class
test_config = {
'toast.one_sided': False,
'toast.two_sided': True
}
def test_burnt_toast(self):
# These values will apply for only this test
self.set_config('toast.one_sided', True)
self.set_config('toast.two_sided', False)
self.toast = Toast()
#...test continues...Charlatan is another open sourced library from Uber for managing fixtures. The following patterns provide a way to integrate the charlatan fixture model into a unittesting framework.
First add a filepath to your fixtures to your configuration file: .Fixture configuration
{
"testing": {
"fixture_filepath": "cerebro/tests/fixtures.yaml"
}
}Charlatan only needs to load the fixtures once, so it makes sense to do it when the package is setup. Adding this to the setup_package() method in the project’s test/init.py accomplishes this when using nosetests. The following is an example from Uber’s Cerebro project:
test/__init__.pyfrom __future__ import absolute_import
import charlatan
import os
import clay
#Get project path
if os.environ.get("CEREBRO_HOME"):
#Get absolute project path from environmental variables
CEREBRO_PATH = os.path.join(os.environ["CEREBRO_HOME"], "cerebro")
else:
#Get project path from relative path
CEREBRO_PATH = os.path.normpath(os.path.abspath(__file__) + "../../../..")
def setup_package():
"""Set up the environment for the whole test package.
Put here all the configuration that needs to be run only once.
"""
#Import fixtures
if clay.config.get('testing.fixture_filepath'):
#Get fixture filepath
if str(CEREBRO_PATH) not in str(clay.config.get('testing.fixture_filepath')):
fixtures_filepath = os.path.join(CEREBRO_PATH, clay.config.get('testing.fixture_filepath'))
else:
fixtures_filepath = clay.config.get('testing.fixture_filepath')
#Load fixtures
charlatan.load(fixtures_filepath,
models_package="cerebro.model",
db_session=db_session) # db_session is a sqlalchemy Session object, for saving fixtures
def teardown_package():
passOnce the fixtures have been loaded, add charlatan’s FixturesManagerMixin to your testcase to allow each test to access and manipulate fixtures. Generally, it saves time if each test can optionally specify a list of fixtures to be installed before each test; adding a few lines to _pre_setup takes care of the installation.
class CerebroTestCase(unittest.TestCase, charlatan.FixturesManagerMixin):
def __call__(self):
#Do pre-test stuff
self.config_copy = clay.config.CONFIG.config.copy()
self._pre_setup()
#Patch config dictionary so tests can customize
with mock.patch.dict(clay.config.CONFIG.config, self.config_copy, clear=True):
unittest.TestCase.__call__(self, result)
#Do post-test stuff
self._post_teardown()
def _pre_setup(self):
self.clean_fixtures_cache()
if hasattr(self, 'test_config'):
#Support ability for tests to have their own test settings
for attr, value in self.test_config.items():
self.set_config(attr, value)
#install class fixtures
if hasattr(self, 'fixtures'):
self.install_fixtures(self.fixtures)
def _post_teardown(self):
self.clean_fixtures_cache()
def set_config(self, key, value):
"""Helper function that tests can call to set config values."""
val = self.config_copy
keys = key.split('.')
for k in keys[:-1]:
val = val[k]
val[keys[-1]] = valueAdditionally, the full charlatan functionality is available in each unit test:
class TestToast(CerebroTestCase):
#These fixtures will be installed with all tests
fixtures = ('toast', 'burnt_toast', 'bread')
def setUp(self):
pass
def tearDown(self):
pass
def test_toastiness(self):
is_burnt = self.toast.burnt # installed fixtures can be accessed
self.install_fixture('wheat_bread') # installs wheat_bread fixture for just this test.
toaster = self.get_fixture('toaster') # can get fixtures manipulate them
toaster.num_slots = 4
toaster.save()
#...test continuesWhen testing, its wise to provide each test with a database containing the correct schema, but no actual data, allowing each test to manipulate the database as necessary. This pattern involves leveraging the transaction capability of sqlalchemy’s scoped_session to accopmlish this goal.
First, add a sqlalchemy configuration for your test database to clay’s configuration file:
{
"testing": {
"fixture_filepath": "cerebro/tests/fixtures.yaml",
"database": {
"sqlalchemy.url": "postgresql://user:password@localhost:1234/test_database",
"echo": True
}
}
}Then, have the engine start when the test package is started by adding to the setup_package() function started here. The following is an example from Uber’s Cerebro project:
test/__init__.py with Test Database Enginefrom __future__ import absolute_import
import charlatan
import os
import clay
from sqlalchemy import engine_from_config
#db_session is a sqlalchemy scoped_session object
#initialize_sql is a function to import Cerebro's models and initialize the sqlalchemy mappers
from cerebro.model.basics import db_session, initialize_sql
test_engine = None
#Get project path
if os.environ.get("CEREBRO_HOME"):
#Get absolute project path from environmental variables
CEREBRO_PATH = os.path.join(os.environ["CEREBRO_HOME"], "cerebro")
else:
#Get project path from relative path
CEREBRO_PATH = os.path.normpath(os.path.abspath(__file__) + "../../../..")
def setup_package():
"""Set up the environment for the whole test package.
Put here all the configuration that needs to be run only once.
"""
#Start test engine
global test_engine
test_engine = engine_from_config(clay.config.get('testing.database'))
#Import fixtures
if clay.config.get('testing.fixture_filepath'):
#Get fixture filepath
if str(CEREBRO_PATH) not in str(clay.config.get('testing.fixture_filepath')):
fixtures_filepath = os.path.join(CEREBRO_PATH, clay.config.get('testing.fixture_filepath'))
else:
fixtures_filepath = clay.config.get('testing.fixture_filepath')
#Load fixtures
charlatan.load(fixtures_filepath,
models_package="cerebro.model",
db_session=db_session) # db_session is a sqlalchemy Session object, for saving fixtures
#Initialize SQLalchemy mappers with test engine
initialize_sql(test_engine)
def teardown_package():
passEach test will open its own connection and transaction with the database, and as part of the teardown the entire transaction will be rolled back, effectively giving each test its own copy of the database to interact with. Continuing to add to our base CerebroTestCase class, we add this logic in the _pre_setup and post_teardown methods:
class CerebroTestCase(unittest.TestCase, charlatan.FixturesManagerMixin):
def __call__(self):
#Do pre-test stuff
self.config_copy = clay.config.CONFIG.config.copy()
self._pre_setup()
#Patch config dictionary so tests can customize
with mock.patch.dict(clay.config.CONFIG.config, self.config_copy, clear=True):
unittest.TestCase.__call__(self, result)
#Do post-test stuff
self._post_teardown()
def _pre_setup(self):
from cerebro.tests import test_engine as engine
# Start a new connection
self.connection = engine.connect()
# Begin a non-ORM transaction
self.trans = self.connection.begin()
# Bind the session to the connection
db_session.configure(bind=self.connection)
self.clean_fixtures_cache()
if hasattr(self, 'test_config'):
#Support ability for tests to have their own test settings
for attr, value in self.test_config.items():
self.set_config(attr, value)
#install class fixtures
if hasattr(self, 'fixtures'):
self.install_fixtures(self.fixtures)
def _post_teardown(self):
# Teardown the transaction
if hasattr(self, "connection"):
# Rollback database
self.trans.rollback()
db_session.remove()
# We have to explicitely close the connection
self.connection.close()
del self.connection
self.clean_fixtures_cache()
def set_config(self, key, value):
"""Helper function that tests can call to set config values."""
val = self.config_copy
keys = key.split('.')
for k in keys[:-1]:
val = val[k]
val[keys[-1]] = valueImportant Note: This pattern can not be used for any methods with explicitly roll back a sqlalchemy session, as it will cascade downward and rollback the test’s transaction.
This release, and all future releases of Clay, will adhere to the versioning scheme described at http://semver.org/
In summary, the major version will be incremented for backwards incompatible changes, the minor version will be incremented for feature releases containing backwards compatible changes, and the patch version will be incremented for bugfix releases.
The CLAY_ENVIRONMENT variable should no longer be used to
differentiate production and development environments, but you should
rather use separate files passed to CLAY_CONFIG for this purpose.
In previous releases, if CLAY_ENVIRONMENT=development was specified,
logging and devserver configuration behaved differently. This check
has been replaced with the debug.enabled boolean flag, which
defaults to false. You will generally want this enabled in your
development configuration.
An additional debug.logging boolean has been added that sets the
default log level of logger instances returned from
clay.config.get_logger() to DEBUG, rather than INFO.
Clay now supports arbitrary logging configuration by setting a
logging key in your configuration, with contents adhering to the
dictConfig
schema specified by the Python standard library.
If no logging element is found in your configuration, Clay will
default to logging all messages to stderr at the WARNING level or
above.
In previous releases, if logging.host were specified in the config,
a clay.logger.UDPHandler was initialized and all log messages were
directed to that host. This option is no longer available, and the
UDPHandler must be initialized using a dictConfig.
logging:
version: 1
handlers:
remote:
class: clay.logger.UDPHandler
level: DEBUG
host: logs.example.com
port: 22000
loggers:
root:
level: DEBUG
handlers: remote
The flask.init.import_name configuration option is now optional. If
not specified, the import_name defaults to clayapp.
The clay framework now has internal tests that may be run with
python setup.py test. These tests are contained in the tests/
directory at the top level of the project and utilize the
(webtest
library.