gearman mysql持久化
gearman 创建Mysql持久化队列的方式如下:
启动gearman,命令如下:
gearmand命令:
-b, –backlog=BACKLOG 连接请求队列的最大值
-d, –daemon Daemon 守护进程化
-f, –file-descriptors=FDS 可打开的文件描述符数量
-h, –help
-l, –log-file=FILE Log 日志文件
-L, –listen=ADDRESS 开启监听的地址
-p, –port=PORT 开启监听的端口
-P, –pid-file=FILE File pid file
-r,–protocol=PROTOCOL 使用的协议
-q, –queue-type=QUEUE 持久化队列类型
-t, –threads=THREADS I/O线程数量
-u, –user=USER 进程的有效用户名
libdrizzle Options:
--libdrizzle-host=HOST Host of server.
--libdrizzle-port=PORT Port of server.
--libdrizzle-uds=UDS Unix domain socket for server.
--libdrizzle-user=USER User name for authentication.
--libdrizzle-password=PASSWORD Password for authentication.
--libdrizzle-db=DB Database to use.
--libdrizzle-table=TABLE Table to use.
--libdrizzle-mysql Use MySQL protocol.
libmemcached Options:
--libmemcached-servers=SERVER_LIST List of Memcached servers to use.
libsqlite3 Options:
--libsqlite3-db=DB Database file to use.
--libsqlite3-table=TABLE Table to use.
libpq Options:
--libpq-conninfo=STRING PostgreSQL connection information string.
--libpq-table=TABLE Table to use.
http Options:
--http-port=PORT Port to listen on.
/usr/local/gearman/sbin/gearmand -p 4730 -L 0.0.0.0 --log-file=/tmp/gearmand-4730.log --pid-file=/tmp/gearmand-4730.pid -q MySQL --mysql-host=localhost --mysql-user=root --mysql-db=gearman --verbose DEBUG-d 我自己看gearmand 参数与上面不同:
[root@localhost sbin]# ./gearmand -h
Allowed options: Allowed options: General options:
-b [ --backlog ] arg (=32) Number of backlog
connections for listen.
-d [ --daemon ] Daemon, detach and run in
the background.
--exceptions Enable protocol exceptions
by default.
-f [ --file-descriptors ] arg Number of file descriptors
to allow for the process
(total connections will be
slightly less). Default is
max allowed for user.
-h [ --help ] Print this help menu.
-j [ --job-retries ] arg (=0) Number of attempts to run
the job before the job
server removes it. This is
helpful to ensure a bad
job does not crash all
available workers. Default
is no limit.
--job-handle-prefix arg Prefix used to generate a
job handle string. If not
provided, the default
"H:<host_name>" is used.
--hashtable-buckets arg (=991) Number of buckets in the
internal job hash tables.
The default of 991 works
well for about three
million jobs in queue. If
the number of jobs in the
queue at any time will
exceed three million, use
proportionally larger
values (991 * # of jobs /
3M). For example, to
accomodate 2^32 jobs, use
1733003. This will consume
~26MB of extra memory.
Gearmand cannot support
more than 2^32 jobs in
queue at this time.
--keepalive Enable keepalive on
sockets.
--keepalive-idle arg (=-1) If keepalive is enabled,
set the value for
TCP_KEEPIDLE for systems
that support it. A value
of -1 means that either
the system does not
support it or an error
occurred when trying to
retrieve the default
value.
--keepalive-interval arg (=-1) If keepalive is enabled,
set the value for
TCP_KEEPINTVL for systems
that support it. A value
of -1 means that either
the system does not
support it or an error
occurred when trying to
retrieve the default
value.
--keepalive-count arg (=-1) If keepalive is enabled,
set the value for
TCP_KEEPCNT for systems
that support it. A value
of -1 means that either
the system does not
support it or an error
occurred when trying to
retrieve the default
value.
-l [ --log-file ] arg (=/opt/var/log/gearmand.log) Log file to write errors
and information to. If the
log-file parameter is
specified as 'stderr',
then output will go to
stderr. If 'none', then no
logfile will be generated.
-L [ --listen ] arg Address the server should
listen on. Default is
INADDR_ANY.
-P [ --pid-file ] arg (=/opt/var/gearmand.pid) File to write process ID
out to.
-r [ --protocol ] arg Load protocol module.
-R [ --round-robin ] Assign work in round-robin
order per worker
connection. The default is
to assign work in the
order of functions added
by the worker.
-q [ --queue-type ] arg (=builtin) Persistent queue type to
use.
--config-file arg (=/opt/etc/gearmand.conf) Can be specified with
'@name', too
--syslog Use syslog.
--coredump Whether to create a core
dump for uncaught signals.
-t [ --threads ] arg (=4) Number of I/O threads to
use, 0 means that gearmand
will try to guess the
maximum number it can use.
Default=4.
-u [ --user ] arg Switch to given user after
startup.
--verbose arg (=ERROR) Set verbose level (FATAL,
ALERT, CRITICAL, ERROR,
WARNING, NOTICE, INFO,
DEBUG).
-V [ --version ] Display the version of
gearmand and exit.
-w [ --worker-wakeup ] arg (=0) Number of workers to
wakeup for each job
received. The default is
to wakeup all available
workers. HTTP:
--http-port arg (=8080) Port to listen on. Gear:
-p [ --port ] arg (=4730) Port the server should listen on.
--ssl Enable ssl connections.
--ssl-ca-file arg CA file.
--ssl-certificate arg SSL certificate.
--ssl-key arg SSL key for certificate. builtin: MySQL:
--mysql-host arg (=localhost) MySQL host.
--mysql-port arg (=3306) Port of server. (by default 3306)
--mysql-user arg MySQL user.
--mysql-password arg MySQL user password.
--mysql-db arg MySQL database.
--mysql-table arg (=gearman_queue) MySQL table name.
/sbin/gearmand -p 4730 -L 0.0.0.0 -p 4730 -q MySQL --mysql-host=localhost --mysql-port 3306
--mysql-user=root --mysql-password pwd --mysql-db=gearman --verbose DEBUG -d 为了安全起见,可以把-L指定为127.0.0.0
执行上面的错误:./gearmand: Error while initializing the queue : MySQL
具体原因查看日志:gearmand.log。显示:
DEBUG 2014-12-05 06:49:31.300168 [ main ] THREADS: 4 -> libgearman-server/gearmand.cc:263
INFO 2014-12-05 06:49:31.300259 [ main ] Initializing MySQL module
ERROR 2014-12-05 06:49:31.341972 [ main ] Failed to connect to database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) -> libgearman-server/plugins/queue/mysql/queue.cc:228
ERROR 2014-12-05 06:49:31.342008 [ main ] Failed to initialize mysql: initialize(QUEUE_ERROR) -> libgearman-server/queue.cc:246
DEBUG 2014-12-05 06:49:31.342081 [ main ] Shutting down all threads -> libgearman-server/gearmand.cc:276
DEBUG 2014-12-05 06:49:31.342093 [ main ] MySQL queue flush -> libgearman-server/plugins/queue/mysql/queue.cc:370
DEBUG 2014-12-05 06:49:31.342103 [ main ] removing queue: FUNCTION -> libgearman-server/gearmand.cc:178
INFO 2014-12-05 06:49:31.342112 [ main ] Shutdown complete
[root@localhost log]# pwd
我去/var/lib/mysql/这个目录没有mysql.sock这个文件。
由于mysql 默认的mysql.sock 是在/var/lib/mysql/mysql.sock,创建符号连接:、(参考文章:
Can't connect to local MySQL server through socket '/tmp/mysql.sock' (11)
解决不能通过mysql .sock连接MySQL问题 这个问题主要提示是,不能通过 '/tmp/mysql .sock'连到服务器,而php标准配置正是用过'/tmp/mysql .sock',但是一些mysql 安装方法 将 mysql .sock放在/var/lib/mysql .sock或者其他的什么地方,你可以通过修改/etc/my.cnf文件来修正它,打开文件,可以看到如下的东东:
[mysql d]
socket=/var/lib/mysql .sock
改一下就好了,但也会引起其他的问题,如mysql 程序连不上了,再加一点:
[mysql ]
socket=/tmp/mysql .sock
或者还可以通过修改php.ini中的配置来使php用其他的mysql .sock来连,这个大家自己去找找
或者用这样的方法:
ln -s /var/lib/mysql /mysql .sock /tmp/mysql .sock 成功了,就是这样ln -s /var/lib/mysql /mysql .sock /tmp/mysql .sock )
locate mysql.sock所在目录。然后创建连接。
ln -s /data/xx/data/mysql.sock /var/lib/mysql/mysql.sock
在 /var/lib/mysql/创建连接后,再运行,果然成功。成功自动创建gearman_queue表。表结构:
create table `gearman_queue` (
`unique_key` varchar(64) NOT NULL,
`function_name` varchar(255) NOT NULL,
`priority` int(11) NOT NULL,
`data` LONGBLOB NOT NULL,
`when_to_run` INT, PRIMARY KEY (`unique_key`)
)
具体的参数,根据自己的服务器状况去修改就行了。
3. 再次登入mysql命令行,执行:
use gearman;show tables;
就可以看到下面多了个“gearman_queue”的表。
这样,gearman就变成了持久化的方式。
=============================================================================
gearman用mysql持久化之后,其实会带来一些问题。
1. 每个任务都会写入数据库,这样会带来磁盘IO的损耗,并且gearman的性能瓶颈又多了一个可能,就是由数据库引起的性能问题。
2. mysql有个“wait_timeout”的参数,在mysql命令行中运行
show variables like "%timeout%";
可以看到wait_timeout的值,默认是28800。也就是说,如果一个mysql的连接,超过28800s没有任何响应,就会断开。
3. gearman持久化的方式,如果超过了mysql的wait_timeouts的时间没有任何响应,和数据库的连接就会被mysql断开,而且gearman目前是没有mysql重新连接的,结果就是,会导致如下错误,必须重启gearman才能重新正常工作。
gearman报错
ERROR 2014-04-01 02:10:02.897899 [ proc ] mysql_stmt_execute failed: -> libgearman-server/plugins/queue/mysql/queue.cc:357
ERROR 2014-04-01 02:10:02.897910 [ proc ] gearman_server_job_add gearman_server_run_command(QUEUE_ERROR) -> libgearman-server/server.cc:301
所以,gearman持久化方式带来的缺点显而易见,该方式下,要避免gearman对mysql连接超时断开,可以将mysql的wait_timeout参数改大。
或者,直接放弃用mysql的持久化方式。
持久化队列将后台作业存放在一个外部持久的队列中。持久化队列只对后台jobs有效,因为前台jobs依附于客户端。如果job服务器挡掉了,客户端会检测到,将会从其他地方重新启动这个前台job或者返回错误。而后台jobs没有依附于客户端,如果要想让它运行则需要提交。
The persistent queue works by calling a module callback function right before putting a new job in the internal queue for pending jobs to be run. This allows the module to store the job about to be run in some persistent way so that it can later be replayed during a restart. Once it is stored through the module, the job is put onto the active runnable queue, waking up available workers if needed. Once the job has been successfully completed by a worker, another module callback function is called to notify the module the job is done and can be removed. If a job server crashes or is restarted between these two calls for a job, the jobs are reloaded during the next job server start. When the job server starts up, it will call a replay callback function in the module to provide a list of all jobs that were not complete. This is used to populate the internal memory queue of jobs to be run. Once this replay is complete, the job server finishes its initialization and the jobs are now runnable once workers connect (the queue should be in the same state as when it crashed). These jobs are removed from the persistent queue when completed as normal. NOTE: Deleting jobs from the persistent queue storage will not remove them from the in-memory queue while the server is running.
创建一个后台job

执行队列中的job:
./gearman -f sayHello -w

http://www.ttlsa.com/gearman/gearman-mysql/
http://huoding.com/2012/10/30/196
更多:
http://www.linuxeye.com/database/mysql-replication-to-redis-by-gearman.html
gearman mysql持久化的更多相关文章
- gearman的持久化,以mysql的方式
1.为什么要持久化? gearman的job server中的工作队列存储在内存中,一旦服务器有未处理的任务时重启或者宕机,那么这些任务就会丢失.持久化存储队列可以允许添加后台任务,并将其存储在外部的 ...
- MySQL 持久化保障机制-redo 日志
我们在 聊一聊 MySQL 中的事务及其实现原理 中提到了 redo 日志,redo 日志是用来保证 MySQL 持久化功能的,需要注意的是 redo 日志是 InnoDB 引擎特有的功能. 为什么 ...
- 在这个应用中,我使用了 MQ 来处理异步流程、Redis 缓存热点数据、MySQL 持久化数据,还有就是在系统中调用另外一个业务系统的接口,对我的应用来说这些都是属于 RPC 调用,而 MQ、MySQL 持久化的数据也会存在于一个分布式文件系统中,他们之间的调用也是需要用 RPC 来完成数据交互的。
在这个应用中,我使用了 MQ 来处理异步流程.Redis 缓存热点数据.MySQL 持久化数据,还有就是在系统中调用另外一个业务系统的接口,对我的应用来说这些都是属于 RPC 调用,而 MQ.MySQ ...
- gearman mysql udf
gearman安装 apt-get install gearman gearman-server libgearman-dev 配置bindip /etc/defalut/gearman-job-se ...
- actor mysql 持久化之 specified actor
持久化到mysql,要求一次操作涉及到的多次读写的事务性.使用的 library 是 postgresql-async, akka 版本是 2.11. 1. 实现 per-user 逻辑,简单来讲,就 ...
- Windows搭建SkyWalking8.3环境进行JAVA应用性能监控及入门示例(使用Mysql持久化)
下载SkyWalking 一.下载地址(点击) 选择tar 解压后进入config文件夹先配置一下Mysql数据源,打开application.yml文件 默认selector是h2我们改成mysql ...
- PDO之MySql持久化自动重连导致内存溢出
前言 最近项目需要一个常驻内存的脚本来执行队列程序,脚本完成后发现Mysql自动重连部分存在内存溢出,导致运行一段时间后,会超出PHP内存限制退出 排查 发现脚本存在内存溢出后排查了一遍代码,基本确认 ...
- 通过Gearman实现MySQL到Redis的数据同步
对于变化频率非常快的数据来说,如果还选择传统的静态缓存方式(Memocached.File System等)展示数据,可能在缓存的存取上会有很大的开销,并不能很好的满足需要,而Redis这样基于内存的 ...
- Gearman + Nodejs + MySQL UDF异步实现 MySQL 到 Redis 的数据同步
[TOC] 1, 环境 CentOS, MySQL, Redis, Nodejs 2, Redis简介 Redis是一个开源的K-V内存数据库,它的key可以是string/set/hash/list ...
随机推荐
- Creating Dialogbased Windows Application (3) / 创建基于对话框的Windows应用程序(三)Checkbox的应用、窗体置顶、设置图标 / VC++, Windows
创建基于对话框的Windows应用程序(三) —— Checkbox的应用.窗体置顶.设置图标 上一节创建的窗体应用程序中,我们用到了Button和StaticText这两个控件.这一节中我们将学习使 ...
- MySQL:cannot allocate the memory for the buffer pool
InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use GCC atomic builtins Inno ...
- Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
错误原因:常量.随机或者依赖时区的表达式不能作为分区函数. 解决方法:把ts列换成datetime类型,创建成功. CREATE TABLE T_log( id INT(11) NOT NULL AU ...
- TCP/IP笔记(一)网络基础知识
计算机与网络发展 计算机自诞生伊始,经历了一系列演变与发展.大型通用机计算机.超级计算机.小型机.个人电脑.工作站.便携式电以及现如今的智能手机终端都是这一过程的产物.它们性能逐年增强,价格却逐年下降 ...
- Python常见经典 python中if __name__ == '__main__': 的解析
当你打开一个.py文件时,经常会在代码的最下面看到if __name__ == '__main__':,现在就来介 绍一下它的作用. 模块是对象,并且所有的模块都有一个内置属性 __name__.一个 ...
- Crontab使用方式
Liunx系统的定时任务需要Crontab来完成 一.添加 添加定时脚本 crontab -e 或者直接编辑/etc/crontab文件进行任务添加 vim /etc/crontab 二.格式 三.举 ...
- Linux Linux程序练习一
注意:在Linux中,"*"是一个通配符,代表所有字符,所以"*"必须要使用转义字符"\"
- Spring4 MVC REST服务使用@RestController实例
在这篇文章中,我们将通过开发使用 Spring4 @RestController 注解来开发基于Spring MVC4的REST风格的JSON服务.我们将扩展这个例子通过简单的注释与JAXB标注域类支 ...
- Google Careers 程序员必修课
quote from : https://www.google.com/about/careers/students/guide-to-technical-development.html Techn ...
- Tomcat工作原理解析!
Tomcat简介 作者:杨晓(http://blog.sina.com.cn/u/1237288325) 一.Tomcat背景 自从JSP发布之后,推出了各式各样的JSP引擎.Apache Gro ...