Why is PuppetDB not answering?

If you rely on PuppetDB, and PuppetDB is not running, you may see this when the agent attempts to run:

$
root@client:~# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 400 on SERVER: Could not retrieve facts for client: Failed to find facts from PuppetDB at client:8081: Connection refused - connect(2)
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for client to PuppetDB at puppet:8081: Connection refused - connect(2)
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
root@client:~# telnet client 8081

One possible cause is that PuppetDB’s database, based on KahaDB, is corrupted. This will appear in the logs with these messages:

$
2014-06-29 21:45:09,060 INFO
Starting broker
2014-06-29 21:45:09,586 INFO [o.a.k.j.Journal] Corrupt journal records found in '/var/lib/puppetdb/mq/localhost/KahaDB
/db-104.log' between offsets: 0-2854479
2014-06-29 21:45:09,779 INFO [o.a.k.j.Journal] Corrupt journal records found in '/var/lib/puppetdb/mq/localhost/KahaDB
/db-104.log' between offsets: 2854480-23709273

Database corruption can occur if you let your PuppetDB server’s disk fill up. If that’s what happened, free up some disk space, then:

How do I fix it?

$
root@puppet:/var/lib/puppet/reports# service puppetdb stop
root@puppet:/var/lib/puppet/reports# cd /var/lib/puppetdb/mq/localhost
root@puppet:/var/lib/puppetdb/mq/localhost# ls -l
total 8
drwxr-xr-x 2 puppetdb puppetdb 4096 Jun 29 21:52 KahaDB
drwxr-xr-x 2 puppetdb puppetdb 4096 Jun 29 21:52 scheduler
root@puppet:/var/lib/puppetdb/mq/localhost# mv KahaDB KahaDB.old
#
root@puppet:/var/lib/puppetdb/mq/localhost# service postgresql restart
* Restarting PostgreSQL 9.3 database server [ OK ]
root@puppet:/var/lib/puppetdb/mq/localhost#
root@puppet:/var/lib/puppetdb/mq/localhost# service puppetdb start
root@puppet:/var/lib/puppetdb/mq/localhost#

..and try again..

$
root@client:~# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for client
Info: Applying configuration version '1435615076'
.
.
.
Share: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Twitter
  • Facebook
  • Google Bookmarks
  • LinkedIn
  • Reddit
  • StumbleUpon

Leave a Reply

Your email address will not be published. Required fields are marked *