Skip to content

Commit

Permalink
api: add Tarantool 3 roles
Browse files Browse the repository at this point in the history
* The role `roles.sharded-queue-router` communicates with storages as
a vshard router and contains the queue API.
* The role `roles.sharded-queue-storage` stores and manages information
about tubes on vshard storages.

Closes #68
  • Loading branch information
oleg-jukovec committed Apr 9, 2024
1 parent 9f2ceed commit 57ce240
Show file tree
Hide file tree
Showing 36 changed files with 1,183 additions and 370 deletions.
66 changes: 53 additions & 13 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,34 +18,74 @@ jobs:
fail-fast: false
matrix:
tarantool:
- '1.10'
- '2.8'
metrics:
- ''
- '1.0.0'
- '2.10'
cartridge: [true]
coveralls: [false]
metrics: [false]
include:
- tarantool: '2.11'
metrics: '1.0.0'
cartridge: true
coveralls: true
metrics: true
- tarantool: 'master'
cartridge: false
coveralls: false
metrics: false

runs-on: ubuntu-20.04
steps:
- name: Clone the module
uses: actions/checkout@v3

- name: Setup tt
run: |
curl -L https://tarantool.io/release/2/installer.sh | sudo bash
sudo apt install -y tt
- name: Install tarantool ${{ matrix.tarantool }}
uses: tarantool/setup-tarantool@v2
uses: tarantool/setup-tarantool@v3
with:
tarantool-version: ${{ matrix.tarantool }}
if: matrix.tarantool != 'master'

- name: Clone the module
uses: actions/checkout@v3
- name: Get Tarantool master latest commit
if: matrix.tarantool == 'master'
run: |
commit_hash=$(git ls-remote https://github.com/tarantool/tarantool.git --branch master | head -c 8)
echo "LATEST_COMMIT=${commit_hash}" >> $GITHUB_ENV
shell: bash

- name: Install requirements
run: make deps
- name: Cache Tarantool master
if: matrix.tarantool == 'master'
id: cache-latest
uses: actions/cache@v3
with:
path: "${GITHUB_WORKSPACE}/bin"
key: cache-latest-${{ env.LATEST_COMMIT }}

- name: Setup Tarantool CE (master)
if: matrix.tarantool == 'master' && steps.cache-latest.outputs.cache-hit != 'true'
run: |
tt init
tt install tarantool master
- name: Install metrics
if: matrix.metrics != ''
- name: Add tt Tarantool to PATH
if: matrix.tarantool == 'master'
run: echo "${GITHUB_WORKSPACE}/bin" >> $GITHUB_PATH

- name: Install requirements
run: |
tarantoolctl rocks install metrics ${{ matrix.metrics }}
tarantool --version
make deps
- name: Install Cartridge requirements
run: make deps-cartridge
if: ${{ matrix.cartridge }}

- name: Install metrics requirements
run: make deps-metrics
if: ${{ matrix.metrics }}

- name: Run linter
run: make lint
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
`tasks` table.
The metric contains labels in the following format:
`{name = "tube_name", state = "task_state"}`
- Role `roles.shareded-queue-router` for Tarantool 3 (#68).
- Role `roles.shareded-queue-storage` for Tarantool 3 (#68).

### Changed

Expand All @@ -44,6 +46,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
`tnt_sharded_queue_api_statistics_tasks` (#71). The metric now has labels
in the format `{name = "tube_name", state = "task_state"}` instead of
`{name = "tube_name", status = "task_state"}`.
- The dependency `cartridge` is removed from the `rockspec` since the module
does not require it to work with Tarantool 3 (#68).

### Fixed

Expand Down
14 changes: 9 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,19 @@ build:

.PHONY: deps
deps:
$(TTCTL) rocks install cartridge 2.8.0
$(TTCTL) rocks install vshard 0.1.26
$(TTCTL) rocks install luacheck 0.26.0
$(TTCTL) rocks install luacov 0.13.0
$(TTCTL) rocks install luacov-coveralls 0.2.3-1 --server=http://luarocks.org
$(TTCTL) rocks install luatest 0.5.7
$(TTCTL) rocks install luatest 1.0.1

.PHONY: deps-full
deps-full: deps
tarantoolctl rocks install metrics 1.0.0
.PHONY: deps-cartridge
deps-cartridge: deps-cartridge
$(TTCTL) rocks install cartridge 2.9.0

.PHONY: deps-metrics
deps-metrics: deps-metrics
$(TTCTL) rocks install metrics 1.0.0

.PHONY: lint
lint:
Expand Down
61 changes: 42 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

# Tarantool Sharded Queue Application

This module provides cartridge roles implementing of a distributed queue compatible with [Tarantool queue](https://github.com/tarantool/queue) (*fifiottl driver*)
This module provides roles for the Tarantool 3 and for the Tarantool
Cartridge implementing of a distributed queue compatible with [Tarantool queue](https://github.com/tarantool/queue) (*fifiottl driver*)

```mermaid
flowchart TB
Expand Down Expand Up @@ -50,9 +50,28 @@ flowchart TB
end
```

## Usage in a cartridge application
## Usage in a Tarantool 3 application

You need to install the Tarantool 3.0.2+ or 3.1+.

1. Add a dependency to your application rockspec.
2. Enable `roles.sharded-queue-router` role on all sharding `router` instances.
3. Enable `roles.sharded-queue-storage` role on all sharding `storage`
instances.
4. Configure tubes for both roles.
5. Do not forget to bootstrap the vshard in your application. See
[init.lua](./init.lua) as an example.

You could see a full example of the configuration in the [config.yaml](./config.yaml).

1. Add dependency to your application rockspec
Be careful, it is impossible to create or drop tubes dynamically by API calls
with Tarantool 3. You need to update the role configuration instead.

## Usage in a Tarantool Cartridge application

1. Add a dependency to your application rockspec. You need to make sure that
the dependency of the Tarantool Cartridge 2.9.0 is enabled because the
`sharded-queue` does not have it by default.
2. Add roles to your application:
```init.lua
cartridge.cfg({
Expand All @@ -74,13 +93,13 @@ cartridge.cfg({
## Usage as a ready-to-deploy service

Prepare `tt` environment:
```
```shell
tt init
git clone https://github.com/tarantool/sharded-queue instances.enabled/sharded-queue
```

Run:
```
```shell
tt pack --app-list sharded-queue rpm --version 1.0.0
```

Expand All @@ -90,10 +109,10 @@ For more details refer to [tt](https://github.com/tarantool/tt/)

## Usage from client perspective

The good old queue api is located on all instances of the router masters that we launched.
For a test configuration, this is one router on `localhost:3301`
The good old queue api is located on all instances of the router masters that
we launched. For a test configuration, this is one router on `localhost:3301`

```
```shell
tarantool@user:~/sharded_queue$ tarantool
Tarantool 1.10.3-6-gfbf53b9
type 'help' for interactive help
Expand All @@ -117,7 +136,7 @@ tarantool> queue_conn:call('queue.tube.test_tube:take')

```
You may also set up tubes using cluster-wide config:
You may also set up tubes using Cartridge cluster-wide config:
```config.yml
tubes:
tube_1:
Expand All @@ -136,36 +155,37 @@ supported:
* `metrics` - enable or disable stats collection by metrics.
metrics >= 0.11.0 is required. It is enabled by default.
## Running locally (as an example)
## Running locally with Tarantool 3 (as an example)
Prepare `tt` environment:
```
```shell
tt init
git clone https://github.com/tarantool/sharded-queue instances.enabled/sharded-queue
```
Install dependencies:
```
```shell
tt build sharded-queue
```
Start default configuration:
```
```shell
tt start sharded-queue
tt cartridge replicasets setup --bootstrap-vshard --name sharded-queue
```
To stop, say:
```
```shell
tt stop sharded-queue
```
## Launching tests
Say:
```
```shell
make deps
make deps-cartridge # For Tarantool < 3.
make deps-metrics # For Tarantool < 3.
make test
```
Expand All @@ -174,7 +194,7 @@ make test
The module exports several metrics if the module `metrics` >= 0.11 is
installed and the feature is not disabled by the configuration.
### Role sharded_queue.api
### Router (`roles.sharded-queue-router` or `sharded_queue.api` for the Cartridge)
* Metric `tnt_sharded_queue_api_statistics_calls_total` is a counter with
the number of requests broken down by [the type of request][queue-statistics].
Expand Down Expand Up @@ -205,7 +225,7 @@ installed and the feature is not disabled by the configuration.
A list of possible call methods: `put`, `take`, `delete`, `release`, `touch`,
`ack`, `bury`, `kick`, `peek`, `drop`.
### Role sharded_queue.storage
### Storage (`roles.sharded-queue-storage` or `sharded_queue.storage` for the Cartridge)
* Metric `tnt_sharded_queue_storage_statistics_calls_total` is a counter with
the number of requests broken down by [the type of request][queue-statistics].
Expand Down Expand Up @@ -269,5 +289,8 @@ installed and the feature is not disabled by the configuration.
If you use **fifottl** driver (default), you can log driver's method calls with `log_request` (log router's and storage's operations).
* You can not create or drop tubes by API calls with Tarantool 3. You need
to update the role configuration instead.
[metrics-summary]: https://www.tarantool.io/en/doc/latest/book/monitoring/api_reference/#summary
[queue-statistics]: https://github.com/tarantool/queue?tab=readme-ov-file#getting-statistics
77 changes: 77 additions & 0 deletions config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
credentials:
users:
client:
password: 'secret'
roles: [super]
replicator:
password: 'secret'
roles: [replication]
storage:
password: 'secret'
roles: [sharding]

iproto:
advertise:
peer:
login: replicator
sharding:
login: storage

sharding:
bucket_count: 3000

groups:
storages:
roles: ['roles.sharded-queue-storage']
roles_cfg:
roles.sharded-queue-storage:
cfg:
metrics: true
tubes:
test_tube: {}
sharding:
roles: [storage]
replication:
failover: manual
replicasets:
storage-001:
leader: storage-001-a
instances:
storage-001-a:
iproto:
listen:
- uri: localhost:3302
storage-001-b:
iproto:
listen:
- uri: localhost:3303
storage-002:
leader: storage-002-a
instances:
storage-002-a:
iproto:
listen:
- uri: localhost:3304
storage-002-b:
iproto:
listen:
- uri: localhost:3305
routers:
roles: ['roles.sharded-queue-router']
roles_cfg:
roles.sharded-queue-router:
cfg:
metrics: true
tubes:
test_tube: {}
app:
module: init
sharding:
roles: [router]
replicasets:
router-001:
instances:
router-001-a:
iproto:
listen:
- uri: localhost:3301
Loading

0 comments on commit 57ce240

Please sign in to comment.