Waiting for 9.3 – Background worker processes

On 6th of December, Alvaro Herrera committed patch:

Background worker processes
 
Background workers are postmaster subprocesses that run arbitrary
user-specified code.  They can request shared memory access as well as
backend database connections; or they can just use plain libpq frontend
database connections.
 
Modules listed in shared_preload_libraries can register background
workers in their _PG_init() function; this is early enough that it's not
necessary to provide an extra GUC option, because the necessary extra
resources can be allocated early on.  Modules can install more than one
bgworker, if necessary.
 
Care is taken that these extra processes do not interfere with other
postmaster tasks: only one such process is started on each ServerLoop
iteration.  This means a large number of them could be waiting to be
started up and postmaster is still able to quickly service external
connection requests.  Also, shutdown sequence should not be impacted by
a worker process that's reasonably well behaved (i.e. promptly responds
to termination signals.)
 
The current implementation lets worker processes specify their start
time, i.e. at what point in the server startup process they are to be
started: right after postmaster start (in which case they mustn't ask
for shared memory access), when consistent state has been reached
(useful during recovery in a HOT standby server), or when recovery has
terminated (i.e. when normal backends are allowed).
 
In case of a bgworker crash, actions to take depend on registration
data: if shared memory was requested, then all other connections are
taken down (as well as other bgworkers), just like it were a regular
backend crashing.  The bgworker itself is restarted, too, within a
configurable timeframe (which can be configured to be never).
 
More features to add to this framework can be imagined without much
effort, and have been discussed, but this seems good enough as a useful
unit already.
 
An elementary sample module is supplied.
 
Author: Álvaro Herrera
 
This patch is loosely based on prior patches submitted by KaiGai Kohei,
and unsubmitted code by Simon Riggs.
 
Reviewed by: KaiGai Kohei, Markus Wanner, Andres Freund,
Heikki Linnakangas, Simon Riggs, Amit Kapila

Continue reading Waiting for 9.3 – Background worker processes

What is the point of bouncing?

Some of you might be familiar with pgBouncer project. Some are not. Some understand what/how/why it does, others do not.

This blog post is to have a place where I can point people who have question about how it works, why, and when it makes sense to use it (pgBouncer that is).

Continue reading What is the point of bouncing?

Window, window on the wall …

And maybe not on the wall, but instead in your SQLz, eating your data.

But a bit more seriously. Ever since PostgreSQL 8.4 we have window functions, but still I see people which do not know it or are wary to use it.

That's why I decided to write a piece on window functions. How they work and what they can be used for.

Continue reading Window, window on the wall …

How I Learned to Stop Worrying and Love the Triggers

Some people are afraid of triggers.

Reasons for this are not really understandable for me, but I guess it stems from the fact that these are usually application developers, and not database admins. Or they encountered some kind of problem with triggers, and now they tend to think that triggers are inherently evil.

But they are not.

As virtually anything, triggers have some benefits, and some drawbacks. With a bit of thinking you can use them to do really cool things. But first you have to understand what exactly trigger is, how it works, and when to use which kind.

Continue reading How I Learned to Stop Worrying and Love the Triggers

Birthday cake again

Last year I got cake in shape of Jameson cake. This year, I was surprised by cake being an idea of my dream-car Mitsubishi EVO:

Evo cake

It was fully edible (including 100% chocolate tires, and spoiler), and very good.

To the potential nay-sayers: I know it's not really shape of Evo. But it had “three diamonds" on both front and back, clearly visible “Evo" signs, so it definitely counts 🙂

Tips N’ Tricks – Running your queries from within Vim

I use VIM. For more or less everything. Including writing blogposts.

Usually, when I was working on blogpost about PostgreSQL, I would write an sql file, switch to another console with psql running, run \i, get output, and then copy/paste the results to my blogpost in another vim.

It worked, but wasn't really nice.

Today, I realized that I can do something much smarter.

I can just type in Vim, and then pass the data to psql, using simple “visual mapping":

:vmap R :!psql -e<enter>

How does it work? When I'm in Vim, and I select (visual) some text, I press shift-R, and the selected blob is sent to psql.

Of course – psql has to know which database to connect to, as which user, and so on, but this is handled by setting PG* environment variables before running Vim.

Thanks to “-e" option, I get all the queries printed back to me, so I don't lose them from my text file.

It works just great.

While I didn't show it in the ascii cast, I can of course also run in this way multiple queries, use transactions, and everything else. The only problem might be that every such run is executed in new psql, which means that you don't have single session.

But, that doesn't seem to be big problem (at least for me).

It would be nice to have vim as full blown sql client, and I think it's perfectly possible, but I just don't care enough to spend time writing necessary scripts.

Getting top-N rows per group

Yesterday on irc someone asked:

Hi, how do I get top 5 values from a column group by another column??

From further discussion, I learned that:

total rows in table is 2 million. It'll have unique words of less than 1 million.. (approx count)

I didn't have time yesterday, but decided to write a solution, or two, to the problem.

Continue reading Getting top-N rows per group

Concurrent REINDEX of all indexes in database

Recent release of new versions of PostgreSQL suggests that you do reindex of all indexes. But this will take a while, and since we don't actually have ‘REINDEX CONCURRENTLY' command – it's a bit tricky.

So, since I will be doing this on several databases, decided to write a script that will handle the work for me.

Continue reading Concurrent REINDEX of all indexes in database