九零不老心
发布于 2024-11-26 / 46 阅读 / 0 评论 / 0 点赞

Flask-SQLAlchemy使用SQLALCHEMY_BINDS时pool_size无效

环境

Flask==2.2.2
Flask-SQLAlchemy==3.0.2

部分配置信息

class Config_base(object):
    # 开启debug 生成模式请改成False
    DEBUG = False
    # log lever : DEBUG INFO WARNING ERROR FATAL
    LOGLEVEL = 'DEBUG'
    # 设置session密钥
    SECRET_KEY = ''
    # mysql
    SQLALCHEMY_DATABASE_URI = "mysql://rpa_mysql:"+urllib.parse.quote_plus(
        "123456")+"@192.168.1.1:3306/test_db"
    # binds
    SQLALCHEMY_BINDS = {
        "doris-btg_marketing": "mysql://test:"+urllib.parse.quote_plus("123456")+"@192.168.1.1:9030/btg_marketing",
    }
    SQLALCHEMY_ECHO = False
    SQLALCHEMY_RECORD_QUERIES = False
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    FLASKY_DB_QUERY_TIMEOUT = 1
    SQLALCHEMY_RECORD_QUERIES = True
    SQLALCHEMY_COMMIT_ON_TEARDOWN = True
    SQLALCHEMY_POOL_SIZE = 180
    SQLALCHEMY_MAX_OVERFLOW = 20
    SQLALCHEMY_POOL_TIMEOUT = 30
    SQLALCHEMY_POOL_RECYCLE = 3600
    SQLALCHEMY_ENGINE_OPTIONS = {
        'pool_size': 80, # 连接池大小
        'max_overflow': 20, # 连接池允许的最大溢出连接数
        'pool_timeout': 30, # 连接池获取连接超时时间(秒)
        'pool_recycle': 3600, # 连接池连接的生命周期(秒),在这之后连接会被回收
        'pool_pre_ping': True # 启用连接池的预检功能,可以检测连接是否可用
    }
    # Flask-APScheduler
    # 一定要开启API功能,这样才可以用api的方式去查看和修改定时任务
    SCHEDULER_API_ENABLED = True
    # 设置时区,时区不一致会导致定时任务的时间错误
    SCHEDULER_TIMEZONE = 'Asia/Shanghai'
    # api前缀(默认是/scheduler)
    SCHEDULER_API_PREFIX = '/scheduler'
    # 配置允许执行定时任务的主机名
    SCHEDULER_ALLOWED_HOSTS = ['*']

故障现象

明明设置了180的连接池,多线程并发10个post请求函数(同时并发写入数据库)时,报错如下:

QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)

初步诊断,SQLALCHEMY_POOL_SIZE 还是SQLALCHEMY_ENGINE_OPTIONS 设置未生效

然后google搜索相关内容:

  1. SQLALCHEMY_ENGINE_OPTIONS only affects the default bind SQLALCHEMY_DATABASE_URI

  2. Values in SQLALCHEMY_BINDS can be a dict with engine options for that bind, including the "url" key instead of only the connection string

  3. https://flask-sqlalchemy.palletsprojects.com/en/stable/binds/#binds

  4. github相关内容https://github.com/pallets-eco/flask-sqlalchemy/issues/799

  5. github相关内容https://github.com/pallets-eco/flask-sqlalchemy/issues/783

解决办法

class Config_base修改SQLALCHEMY_BINDS(SQLALCHEMY_ENGINE_OPTIONS只对默认的SQLALCHEMY_DATABASE_URI生效,binds数据库需要SQLALCHEMY_BIND中单独指定参数)

    # binds
    SQLALCHEMY_BINDS = {
        "doris-btg_marketing": {'url':"mysql://test:"+urllib.parse.quote_plus("123456")+"@192.168.1.1:9030/btg_marketing",'pool_size': 120,},
    }