| title | keywords | description | |||
|---|---|---|---|---|---|
FAQ |
|
This article lists solutions to common problems when using Apache APISIX. |
As organizations move towards cloud native microservices, there is a need for an API gateway that is performant, flexible, secure and scalable.
APISIX outperforms other API gateways in these metrics while being platform agnostic and fully dynamic delivering features like supporting multiple protocols, fine-grained routing and multi-language support.
Apache APISIX differs in the following ways:
- It uses etcd to save and synchronize configurations rather than relational databases like PostgreSQL or MySQL. The real-time event notification system in etcd is easier to scale than in these alternatives. This allows APISIX to synchronize the configuration in real-time, makes the code concise and avoids a single point of failure.
- Fully dynamic.
- Supports hot loading of Plugins.
Apache APISIX delivers the best performance among other API gateways with a single-core QPS of 18,000 with an average delay of 0.2 ms.
Specific results of the performance benchmarks can be found here.
Apache APISIX is platform agnostic and avoids vendor lock-in. It is built for cloud native environments and can run on bare-metal machines to Kubernetes. It even support Apple Silicon chips.
Apache APISIX is fully dynamic in the sense that it doesn't require restarts to change its behavior.
It does the following dynamically:
- Reloading Plugins
- Proxy rewrites
- Proxy mirror
- Response rewrites
- Health checks
- Traffic split
Yes. Apache APISIX has an experimental feature called Apache APISIX Dashboard, which is independent from Apache APISIX. To work with Apache APISIX through a user interface, you can deploy the Apache APISIX Dashboard.
Yes. Apache APISIX is flexible and extensible through the use of custom Plugins that can be specific to user needs.
You can write your own Plugins by referring to How to write your own Plugins.
In addition to the basic functionality of storing the configurations, Apache APISIX also needs a storage system that supports these features:
- Distributed deployments in clusters.
- Guarded transactions by comparisons.
- Multi-version concurrency control.
- Notifications and watch streams.
- High performance with minimum read/write latency.
etcd provides these features and more making it ideal over other databases like PostgreSQL and MySQL.
To learn more on how etcd compares with other alternatives see this comparison chart.
When installing Apache APISIX dependencies with LuaRocks, why does it cause a timeout or result in a slow or unsuccessful installation?
This is likely because the LuaRocks server used is blocked.
To solve this you can use https_proxy or use the --server flag to specify a faster LuaRocks server.
You can run the command below to see the available servers (needs LuaRocks 3.0+):
luarocks config rocks_serversMainland China users can use luarocks.cn as the LuaRocks server. You can use this wrapper with the Makefile to set this up:
make deps ENV_LUAROCKS_SERVER=https://luarocks.cnIf this does not solve your problem, you can try getting a detailed log by using the --verbose or -v flag to diagnose the problem.
Some functions need to introduce additional NGINX modules, which requires APISIX to run on APISIX-Runtime. If you need these functions, you can refer to the code in api7/apisix-build-tools to build your own APISIX-Runtime environment.
Let's take an example query foo.com/product/index.html?id=204&page=2 and consider that you need to make a gray release based on the id in the query string with this condition:
- Group A:
id <= 1000 - Group B:
id > 1000
There are two different ways to achieve this in Apache APISIX:
- Using the
varsfield in a Route:
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/index.html",
"vars": [
["arg_id", "<=", "1000"]
],
"plugins": {
"redirect": {
"uri": "/test?group_id=1"
}
}
}'
curl -i http://127.0.0.1:9180/apisix/admin/routes/2 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/index.html",
"vars": [
["arg_id", ">", "1000"]
],
"plugins": {
"redirect": {
"uri": "/test?group_id=2"
}
}
}'All the available operators of the current lua-resty-radixtree are listed here.
- Using the traffic-split Plugin.
For example, you need to redirect traffic from http://foo.com to https://foo.com.
Apache APISIX provides several different ways to achieve this:
- Setting
http_to_httpstotruein the redirect Plugin:
curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/hello",
"host": "foo.com",
"plugins": {
"redirect": {
"http_to_https": true
}
}
}'- Advanced routing with
varsin the redirect Plugin:
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/hello",
"host": "foo.com",
"vars": [
[
"scheme",
"==",
"http"
]
],
"plugins": {
"redirect": {
"uri": "https://$host$request_uri",
"ret_code": 301
}
}
}'- Using the
serverlessPlugin:
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/hello",
"plugins": {
"serverless-pre-function": {
"phase": "rewrite",
"functions": ["return function() if ngx.var.scheme == \"http\" and ngx.var.host == \"foo.com\" then ngx.header[\"Location\"] = \"https://foo.com\" .. ngx.var.request_uri; ngx.exit(ngx.HTTP_MOVED_PERMANENTLY); end; end"]
}
}
}'To test this serverless Plugin:
curl -i -H 'Host: foo.com' http://127.0.0.1:9080/helloThe response should be:
HTTP/1.1 301 Moved Permanently
Date: Mon, 18 May 2020 02:56:04 GMT
Content-Type: text/html
Content-Length: 166
Connection: keep-alive
Location: https://foo.com/hello
Server: APISIX web server
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>openresty</center>
</body>
</html>
By default the log level of Apache APISIX is set to warn. You can set this to info to trace the messages printed by core.log.info.
For this, you can set the error_log_level parameter in your configuration file (conf/config.yaml) as shown below and reload Apache APISIX.
nginx_config:
error_log_level: "info"All Plugins in Apache APISIX are hot reloaded.
You can learn more about hot reloading of Plugins here.
By default, Apache APISIX listens only on port 9080 when handling HTTP requests.
To configure Apache APISIX to listen on multiple ports, you can:
-
Modify the parameter
node_listeninconf/config.yaml:apisix: node_listen: - 9080 - 9081 - 9082Similarly for HTTPS requests, modify the parameter
ssl.listeninconf/config.yaml:apisix: ssl: enable: true listen: - port: 9443 - port: 9444 - port: 9445 -
Reload or restart Apache APISIX.
After uploading the SSL certificate, why can't the corresponding route be accessed through HTTPS + IP?
If you directly use HTTPS + IP address to access the server, the server will use the IP address to compare with the bound SNI. Since the SSL certificate is bound to the domain name, the corresponding resource cannot be found in the SNI, so that the certificate will be verified. The authentication fails, and the user cannot access the gateway via HTTPS + IP.
You can implement this function by setting the fallback_sni parameter in the configuration file and configuring the domain name. When the user uses HTTPS + IP to access the gateway, when the SNI is empty, it will fall back to the default SNI to achieve HTTPS + IP access to the gateway.
apisix
ssl：
fallback_sni: "${your sni}"Apache APISIX uses etcd for its configuration center. etcd provides subscription functions like watch and watchdir that can monitor changes to specific keywords or directories.
In Apache APISIX, we use etcd.watchdir to monitor changes in a directory.
If there is no change in the directory being monitored, the process will be blocked until it times out or run into any errors.
If there are changes in the directory being monitored, etcd will return this new data within milliseconds and Apache APISIX will update the cache memory.
By default, Apache APISIX reads the instance id from conf/apisix.uid. If this is not found and no id is configured, Apache APISIX will generate a uuid for the instance id.
To specify a meaningful id to bind Apache APISIX to your internal system, set the id in your conf/config.yaml file:
apisix:
id: "your-id"Why are there errors saying "failed to fetch data from etcd, failed to read etcd dir, etcd key: xxxxxx" in the error.log?
Please follow the troubleshooting steps described below:
-
Make sure that there aren't any networking issues between Apache APISIX and your etcd deployment in your cluster.
-
If your network is healthy, check whether you have enabled the gRPC gateway for etcd. The default state depends on whether you used command line options or a configuration file to start the etcd server.
- If you used command line options, gRPC gateway is enabled by default. You can enable it manually as shown below:
etcd --enable-grpc-gateway --data-dir=/path/to/data
Note: This flag is not shown while running
etcd --help.- If you used a configuration file, gRPC gateway is disabled by default. You can manually enable it as shown below:
In
etcd.json:{ "enable-grpc-gateway": true, "data-dir": "/path/to/data" }In
etcd.conf.yml:enable-grpc-gateway: true
Note: This distinction was eliminated by etcd in their latest master branch but wasn't backported to previous versions.
Apache APISIX can be made highly available by adding a load balancer in front of it as APISIX's data plane is stateless and can be scaled when needed.
The control plane of Apache APISIX is highly available as it relies only on an etcd cluster.
When executing make deps to install Apache APISIX from source, you can get an error as shown below:
$ make deps
......
Error: Failed installing dependency: https://luarocks.org/luasec-0.9-1.src.rock - Could not find header file for OPENSSL
No file openssl/ssl.h in /usr/local/include
You may have to install OPENSSL in your system and/or pass OPENSSL_DIR or OPENSSL_INCDIR to the luarocks command.
Example: luarocks install luasec OPENSSL_DIR=/usr/local
make: *** [deps] Error 1This is caused by the missing OpenResty openssl development kit. To install it, refer installing dependencies.
You can follow the steps below to configure this:
- Configure different ports for Apache APISIX proxy and Admin API. Or, disable the Admin API.
deployment:
admin:
admin_listen: # use a separate port
ip: 127.0.0.1
port: 9180- Add a proxy Route for the Apache APISIX dashboard:
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uris":[ "/*" ],
"name":"apisix_proxy_dashboard",
"upstream":{
"nodes":[
{
"host":"127.0.0.1",
"port":9000,
"weight":1
}
],
"type":"roundrobin"
}
}'Note: The Apache APISIX Dashboard is listening on 127.0.0.1:9000.
You can use the vars field in a Route for matching regular expressions:
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/*",
"vars": [
["uri", "~~", "^/[a-z]+$"]
],
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:1980": 1
}
}
}'And to test this request:
# uri matched
$ curl http://127.0.0.1:9080/hello -i
HTTP/1.1 200 OK
...
# uri didn't match
$ curl http://127.0.0.1:9080/12ab -i
HTTP/1.1 404 Not Found
...For more info on using vars refer to lua-resty-expr.
Does the Upstream node support configuring a FQDN address?
Yes. The example below shows configuring the FQDN httpbin.default.svc.cluster.local (a Kubernetes service):
curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/ip",
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.default.svc.cluster.local": 1
}
}
}'To test this Route:
$ curl http://127.0.0.1:9080/ip -i
HTTP/1.1 200 OK
...X-API-KEY of the Admin API refers to the apisix.admin_key.key in your conf/config.yaml file. It is the access token for the Admin API.
By default, it is set to edd1c9f034335f136f87ad84b625c8f1 and can be modified by changing the parameter in your conf/config.yaml file:
apisix:
admin_key
-
name: "admin"
key: newkey
role: adminNow, to access the Admin API:
$ curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: newkey' -X PUT -d '
{
"uris":[ "/*" ],
"name":"admin-token-test",
"upstream":{
"nodes":[
{
"host":"127.0.0.1",
"port":1980,
"weight":1
}
],
"type":"roundrobin"
}
}'
HTTP/1.1 200 OK
......Note: By using the default token, you could be exposed to security risks. It is required to update it when deploying to a production environment.
By default, Apache APISIX only allows IPs in the range 127.0.0.0/24 to access the Admin API.
To allow IPs in all ranges, you can update your configuration file as show below and restart or reload Apache APISIX.
deployment:
admin:
allow_admin:
- 0.0.0.0/0Note: This should only be used in non-production environments to allow all clients to access Apache APISIX and is not safe for production environments. Always authorize specific IP addresses or address ranges for production environments.
You can run the commands below to achieve this:
curl --output /root/.acme.sh/renew-hook-update-apisix.sh --silent https://gist.githubusercontent.com/anjia0532/9ebf8011322f43e3f5037bc2af3aeaa6/raw/65b359a4eed0ae990f9188c2afa22bacd8471652/renew-hook-update-apisix.shchmod +x /root/.acme.sh/renew-hook-update-apisix.shacme.sh --issue --staging -d demo.domain --renew-hook "/root/.acme.sh/renew-hook-update-apisix.sh -h http://apisix-admin:port -p /root/.acme.sh/demo.domain/demo.domain.cer -k /root/.acme.sh/demo.domain/demo.domain.key -a xxxxxxxxxxxxx"acme.sh --renew --domain demo.domainYou can check this post for a more detailed instruction on setting this up.
To strip a prefix from a path in your route, like to take /foo/get and strip it to /get, you can use the proxy-rewrite Plugin:
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/foo/*",
"plugins": {
"proxy-rewrite": {
"regex_uri": ["^/foo/(.*)","/$1"]
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org:80": 1
}
}
}'And to test this configuration:
curl http://127.0.0.1:9080/foo/get -i
HTTP/1.1 200 OK
...
{
...
"url": "http://127.0.0.1/get"
}You can manually set the path to your certificate by adding it to your conf/config.yaml file as shown below:
apisix:
ssl:
ssl_trusted_certificate: /path/to/certs/ca-certificates.crtNote: When you are trying to connect TLS services with cosocket and if APISIX does not trust the peer's TLS certificate, you should set the parameter apisix.ssl.ssl_trusted_certificate.
For example, if you are using Nacos for service discovery in APISIX, and Nacos has TLS enabled (configured host starts with https://), you should set apisix.ssl.ssl_trusted_certificate and use the same CA certificate as Nacos.
This error is caused by installing Apache APISIX in the /root directory. The worker process would by run by the user "nobody" and it would not have enough permissions to access the files in the /root directory.
To fix this, you can change the APISIX installation directory to the recommended directory: /usr/local.
The differences between the two are described in the table below:
plugin-metadata |
plugin-config |
|---|---|
| Metadata of a Plugin shared by all configuration instances of the Plugin. | Collection of configuration instances of multiple different Plugins. |
| Used when there are property changes that needs to be propagated across all configuration instances of a Plugin. | Used when you need to reuse a common set of configuration instances so that it can be extracted to a plugin-config and bound to different Routes. |
| Takes effect on all the entities bound to the configuration instances of the Plugin. | Takes effect on Routes bound to the plugin-config. |
You can create a route named health-info and enable the fault-injection plugin (where YOUR-TOKEN is the user's token; 127.0.0.1 is the IP address of the control plane, which can be modified by yourself):
curl http://127.0.0.1:9180/apisix/admin/routes/health-info \
-H 'X-API-KEY: YOUR-TOKEN' -X PUT -d '
{
"plugins": {
"fault-injection": {
"abort": {
"http_status": 200,
"body": "fine"
}
}
},
"uri": "/status"
}'Verification:
Access the /status of the Apache APISIX data plane to detect APISIX. If the response code is 200, it means APISIX is alive.
:::note
This method only detects whether the APISIX data plane is alive or not. It does not mean that the routing and other functions of APISIX are normal. These require more routing-level detection.
:::
What are the scenarios with high APISIX latency related to etcd and how to fix them?
etcd is the data storage component of apisix, and its stability is related to the stability of APISIX.
In actual scenarios, if APISIX uses a certificate to connect to etcd through HTTPS, the following two problems of high latency for data query or writing may occur:
- Query or write data through APISIX Admin API.
- In the monitoring scenario, Prometheus crawls the APISIX data plane Metrics API timeout.
These problems related to higher latency seriously affect the service stability of APISIX, and the reason why such problems occur is mainly because etcd provides two modes of operation: HTTP (HTTPS) and gRPC. And APISIX uses the HTTP (HTTPS) protocol to operate etcd by default.
In this scenario, etcd has a bug about HTTP/2: if etcd is operated over HTTPS (HTTP is not affected), the upper limit of HTTP/2 connections is the default 250 in Golang. Therefore, when the number of APISIX data plane nodes is large, once the number of connections between all APISIX nodes and etcd exceeds this upper limit, the response of APISIX API interface will be very slow.
In Golang, the default upper limit of HTTP/2 connections is 250, the code is as follows:
package http2
import ...
const (
prefaceTimeout = 10 * time.Second
firstSettingsTimeout = 2 * time.Second // should be in-flight with preface anyway
handlerChunkWriteSize = 4 << 10
defaultMaxStreams = 250 // TODO: make this 100 as the GFE seems to?
maxQueuedControlFrames = 10000
)etcd officially maintains two main branches, 3.4 and 3.5. In the 3.4 series, the recently released 3.4.20 version has fixed this issue. As for the 3.5 version, the official is preparing to release the 3.5.5 version a long time ago, but it has not been released as of now (2022.09.13). So, if you are using etcd version less than 3.5.5, you can refer to the following ways to solve this problem:
- Change the communication method between APISIX and etcd from HTTPS to HTTP.
- Roll back the etcd to
3.4.20. - Clone the etcd source code and compile the
release-3.5branch directly (this branch has fixed the problem of HTTP/2 connections, but the new version has not been released yet).
The way to recompile etcd is as follows:
git checkout release-3.5
make GOOS=linux GOARCH=amd64The compiled binary is in the bin directory, replace it with the etcd binary of your server environment, and then restart etcd:
For more information, please refer to:
- when etcd node have many http long polling connections, it may cause etcd to respond slowly to http requests.
- bug: when apisix starts for a while, its communication with etcd starts to time out
- the prometheus metrics API is tool slow
- Support configuring
MaxConcurrentStreamsfor http2
Another solution is to switch to an experimental gRPC-based configuration synchronization. This requires setting use_grpc: true in the configuration file conf/config.yaml:
etcd:
use_grpc: true
host:
- "http://127.0.0.1:2379"
prefix: "/apisix"If you are using the file-logger plugin but getting garbled logs, one possible reason is your upstream response has returned a compressed response body. You can fix this by setting the accept-encoding in the request header to not receive compressed responses using the proxy-rewirte plugin:
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: YOUR-TOKEN' -X PUT -d '
{
"methods":[
"GET"
],
"uri":"/test/index.html",
"plugins":{
"proxy-rewrite":{
"headers":{
"set":{
"accept-encoding":"gzip;q=0,deflate,sdch"
}
}
}
},
"upstream":{
"type":"roundrobin",
"nodes":{
"127.0.0.1:80":1
}
}
}'Suppose you have an ETCD cluster that enables the auth. To access this cluster, you need to configure the correct username and password for Apache APISIX in conf/config.yaml:
deployment:
etcd:
host:
- "http://127.0.0.1:2379"
user: etcd_user # username for etcd
password: etcd_password # password for etcdFor other ETCD configurations, such as expiration times, retries, and so on, you can refer to the ETCD section in the conf/config-default.yaml file.
What is the difference between SSLs and tls.client_cert in upstream configurations, and ssl_trusted_certificate in config-default.yaml?
The ssls is managed through the /apisix/admin/ssls API. It's used for managing TLS certificates. These certificates may be used during TLS handshake (between Apache APISIX and its clients). Apache APISIX uses Server Name Indication (SNI) to differentiate between certificates of different domains.
The tls.client_cert, tls.client_key, and tls.client_cert_id in upstream are used for mTLS communication with the upstream.
The ssl_trusted_certificate in config-default.yaml configures a trusted CA certificate. It is used for verifying some certificates signed by private authorities within APISIX, to avoid APISIX rejects the certificate. Note that it is not used to trust the certificates of APISIX upstream, because APISIX does not verify the legality of the upstream certificates. Therefore, even if the upstream uses an invalid TLS certificate, it can still be accessed without configuring a root certificate.
You can find more answers on: