#297 If mariadb is restarted, queuedaemons fail

باز‌کردن
1 سال پیش باز شده توسط aab · 1 دیدگاه

For example, take this notice (published shortly after a mariadb restart): https://gnusocial.net/notice/14505811

The notice is not federated to other servers, only published locally. Restarting queuedaemons fixes (or, to be more precise, mitigates) the issue.

GS debug logs:

2023-05-01 19:12:15 LOG_ERR: [gnusocial.net:queuedaemon.php:2532] Error during Notice->getProfileTags() for id==14505811: [Profile:157720] DB_DataObject error []: MDB2 Error: unknown error 2023-05-01 19:12:15 LOG_ERR: [gnusocial.net:queuedaemon.php:2532] Distrib queue exception saving notice 14505811: [Reply] DB_DataObject error []: MDB2 Error: unknown error #0 /var/www/gnusocial/extlib/DB/DataObject.php(1627): Memcached_DataObject->raiseError('MDB2 Error: unk...') #1 /var/www/gnusocial/classes/Memcached_DataObject.php(762): DB_DataObject->_query('SELECT 0 /*M! +...') #2 /var/www/gnusocial/extlib/DB/DataObject.php(3135): Memcached_DataObject->_query('SELECT 0 /*M! +...') #3 /var/www/gnusocial/classes/Memcached_DataObject.php(103): DB_DataObject->query('SELECT 0 /*M! +...') #4 /var/www/gnusocial/classes/Managed_DataObject.php(91): Memcached_DataObject::multiGetClass('Reply', 'notice_id', Array, true, false) #5 /var/www/gnusocial/classes/Notice.php(1867): Managed_DataObject::multiGet('notice_id', Array) #6 /var/www/gnusocial/classes/Notice.php(1905): Notice->getReplies() #7 /var/www/gnusocial/lib/notices/distribqueuehandler.php(81): Notice->sendReplyNotifications() #8 /var/www/gnusocial/plugins/RedisQueue/classes/RedisQueueManager.php(58): DistribQueueHandler->handle(Object(Notice)) #9 /var/www/gnusocial/lib/util/iomaster.php(286): RedisQueueManager->poll() #10 /var/www/gnusocial/lib/util/iomaster.php(160): IoMaster->poll() #11 /var/www/gnusocial/scripts/queuedaemon.php(112): IoMaster->service() #12 /var/www/gnusocial/lib/util/spawningdaemon.php(188): QueueDaemon->runThread() #13 /var/www/gnusocial/lib/util/spawningdaemon.php(82): SpawningDaemon->initAndRunChild(1) #14 /var/www/gnusocial/lib/util/daemon.php(205): SpawningDaemon->run() #15 /var/www/gnusocial/scripts/queuedaemon.php(186): Daemon->runOnce() #16 {main}

2023-05-01 19:12:15 LOG_ERR: [gnusocial.net:queuedaemon.php:2532] Distrib queue exception saving notice 14505811: [Profile:157720] DB_DataObject error []: MDB2 Error: unknown error #0 /var/www/gnusocial/extlib/DB/DataObject.php(1627): Memcached_DataObject->raiseError('MDB2 Error: unk...') #1 /var/www/gnusocial/classes/Memcached_DataObject.php(762): DB_DataObject->_query('SELECT * \n FROM...') #2 /var/www/gnusocial/extlib/DB/DataObject.php(1192): Memcached_DataObject->_query('SELECT * \n FROM...') #3 /var/www/gnusocial/classes/GS_DataObject.php(134): DB_DataObject->find(1) #4 /var/www/gnusocial/extlib/DB/DataObject.php(904): GS_DataObject->find(1) #5 /var/www/gnusocial/classes/GS_DataObject.php(108): DB_DataObject->get('id', 157720) #6 /var/www/gnusocial/classes/Memcached_DataObject.php(49): GS_DataObject->get('id', 157720) #7 /var/www/gnusocial/classes/Managed_DataObject.php(50): Memcached_DataObject::getClassKV('Profile', 'id', 157720) #8 /var/www/gnusocial/classes/Notice.php(151): Managed_DataObject::getKV('id', 157720) #9 /var/www/gnusocial/classes/Notice.php(2897): Notice->getProfile() #10 /var/www/gnusocial/classes/Notice.php(2883): Notice->_inScope(NULL) #11 /var/www/gnusocial/plugins/OStatus/OStatusPlugin.php(142): Notice->inScope(NULL) #12 /var/www/gnusocial/lib/util/event.php(89): OStatusPlugin->onStartEnqueueNotice(Object(Notice), Array) #13 /var/www/gnusocial/lib/util/util.php(1691): Event::handle('StartEnqueueNot...', Array) #14 /var/www/gnusocial/lib/notices/distribqueuehandler.php(100): common_enqueue_notice(Object(Notice)) #15 /var/www/gnusocial/plugins/RedisQueue/classes/RedisQueueManager.php(58): DistribQueueHandler->handle(Object(Notice)) #16 /var/www/gnusocial/lib/util/iomaster.php(286): RedisQueueManager->poll() #17 /var/www/gnusocial/lib/util/iomaster.php(160): IoMaster->poll() #18 /var/www/gnusocial/scripts/queuedaemon.php(112): IoMaster->service() #19 /var/www/gnusocial/lib/util/spawningdaemon.php(188): QueueDaemon->runThread() #20 /var/www/gnusocial/lib/util/spawningdaemon.php(82): SpawningDaemon->initAndRunChild(1) #21 /var/www/gnusocial/lib/util/daemon.php(205): SpawningDaemon->run() #22 /var/www/gnusocial/scripts/queuedaemon.php(186): Daemon->runOnce() #23 {main}

For example, take this notice (published shortly after a mariadb restart): https://gnusocial.net/notice/14505811 The notice is not federated to other servers, only published locally. Restarting queuedaemons fixes (or, to be more precise, mitigates) the issue. GS debug logs: ```2023-05-01 19:12:15 LOG_ERR: [gnusocial.net:queuedaemon.php:2532] Error during Notice->getProfileTags() for id==14505811: [Profile:157720] DB_DataObject error []: MDB2 Error: unknown error 2023-05-01 19:12:15 LOG_ERR: [gnusocial.net:queuedaemon.php:2532] Distrib queue exception saving notice 14505811: [Reply] DB_DataObject error []: MDB2 Error: unknown error #0 /var/www/gnusocial/extlib/DB/DataObject.php(1627): Memcached_DataObject->raiseError('MDB2 Error: unk...') #1 /var/www/gnusocial/classes/Memcached_DataObject.php(762): DB_DataObject->_query('SELECT 0 /*M! +...') #2 /var/www/gnusocial/extlib/DB/DataObject.php(3135): Memcached_DataObject->_query('SELECT 0 /*M! +...') #3 /var/www/gnusocial/classes/Memcached_DataObject.php(103): DB_DataObject->query('SELECT 0 /*M! +...') #4 /var/www/gnusocial/classes/Managed_DataObject.php(91): Memcached_DataObject::multiGetClass('Reply', 'notice_id', Array, true, false) #5 /var/www/gnusocial/classes/Notice.php(1867): Managed_DataObject::multiGet('notice_id', Array) #6 /var/www/gnusocial/classes/Notice.php(1905): Notice->getReplies() #7 /var/www/gnusocial/lib/notices/distribqueuehandler.php(81): Notice->sendReplyNotifications() #8 /var/www/gnusocial/plugins/RedisQueue/classes/RedisQueueManager.php(58): DistribQueueHandler->handle(Object(Notice)) #9 /var/www/gnusocial/lib/util/iomaster.php(286): RedisQueueManager->poll() #10 /var/www/gnusocial/lib/util/iomaster.php(160): IoMaster->poll() #11 /var/www/gnusocial/scripts/queuedaemon.php(112): IoMaster->service() #12 /var/www/gnusocial/lib/util/spawningdaemon.php(188): QueueDaemon->runThread() #13 /var/www/gnusocial/lib/util/spawningdaemon.php(82): SpawningDaemon->initAndRunChild(1) #14 /var/www/gnusocial/lib/util/daemon.php(205): SpawningDaemon->run() #15 /var/www/gnusocial/scripts/queuedaemon.php(186): Daemon->runOnce() #16 {main}``` ```2023-05-01 19:12:15 LOG_ERR: [gnusocial.net:queuedaemon.php:2532] Distrib queue exception saving notice 14505811: [Profile:157720] DB_DataObject error []: MDB2 Error: unknown error #0 /var/www/gnusocial/extlib/DB/DataObject.php(1627): Memcached_DataObject->raiseError('MDB2 Error: unk...') #1 /var/www/gnusocial/classes/Memcached_DataObject.php(762): DB_DataObject->_query('SELECT * \n FROM...') #2 /var/www/gnusocial/extlib/DB/DataObject.php(1192): Memcached_DataObject->_query('SELECT * \n FROM...') #3 /var/www/gnusocial/classes/GS_DataObject.php(134): DB_DataObject->find(1) #4 /var/www/gnusocial/extlib/DB/DataObject.php(904): GS_DataObject->find(1) #5 /var/www/gnusocial/classes/GS_DataObject.php(108): DB_DataObject->get('id', 157720) #6 /var/www/gnusocial/classes/Memcached_DataObject.php(49): GS_DataObject->get('id', 157720) #7 /var/www/gnusocial/classes/Managed_DataObject.php(50): Memcached_DataObject::getClassKV('Profile', 'id', 157720) #8 /var/www/gnusocial/classes/Notice.php(151): Managed_DataObject::getKV('id', 157720) #9 /var/www/gnusocial/classes/Notice.php(2897): Notice->getProfile() #10 /var/www/gnusocial/classes/Notice.php(2883): Notice->_inScope(NULL) #11 /var/www/gnusocial/plugins/OStatus/OStatusPlugin.php(142): Notice->inScope(NULL) #12 /var/www/gnusocial/lib/util/event.php(89): OStatusPlugin->onStartEnqueueNotice(Object(Notice), Array) #13 /var/www/gnusocial/lib/util/util.php(1691): Event::handle('StartEnqueueNot...', Array) #14 /var/www/gnusocial/lib/notices/distribqueuehandler.php(100): common_enqueue_notice(Object(Notice)) #15 /var/www/gnusocial/plugins/RedisQueue/classes/RedisQueueManager.php(58): DistribQueueHandler->handle(Object(Notice)) #16 /var/www/gnusocial/lib/util/iomaster.php(286): RedisQueueManager->poll() #17 /var/www/gnusocial/lib/util/iomaster.php(160): IoMaster->poll() #18 /var/www/gnusocial/scripts/queuedaemon.php(112): IoMaster->service() #19 /var/www/gnusocial/lib/util/spawningdaemon.php(188): QueueDaemon->runThread() #20 /var/www/gnusocial/lib/util/spawningdaemon.php(82): SpawningDaemon->initAndRunChild(1) #21 /var/www/gnusocial/lib/util/daemon.php(205): SpawningDaemon->run() #22 /var/www/gnusocial/scripts/queuedaemon.php(186): Daemon->runOnce() #23 {main}```
aab نظر 1 سال پیش
نویسنده

Ok, not a solution, but it seems to work: including this line in /etc/systemd/system/mysqld.service:

ExecStartPost=/bin/sh /var/www/gnusocial/scripts/stopdaemons.sh

If you have monit installed and configured to boot queuedaemons, the previous line stops them, and monit reboots them within minutes.

Ok, not a solution, but it seems to work: including this line in /etc/systemd/system/mysqld.service: ExecStartPost=/bin/sh /var/www/gnusocial/scripts/stopdaemons.sh If you have monit installed and configured to boot queuedaemons, the previous line stops them, and monit reboots them within minutes.
برای پیوستن به گفتگو، وارد شودید.
بدون نقطه عطف
بدون مسئول رسیدگی
1 مشارکت کننده
درحال بارگذاری...
لغو
ذخيره
هنوز محتوایی ایجاد نشده.