DISCLAIMER

At the time of publication, none of what is discussed below is implemented in Glowdust. Some of it may make it into the product, but everything that follows is design thoughts, not documentation

As the saying goes, transactions are pretty simple until you actually start using them.

I know that isn’t really a saying, but it should be because it’s absolutely true. Transactions are weird beasts, except that they aren’t, really.

Look, it’s complicated.

Transactions are the leaky abstraction of the database world. It’s the thin line between computer science and software engineering.

And they affect every aspect of the system.

You can always skip them, and label your system as OLAP only. Lots of people have had success with this.

But if you want your writes to make sense, you need to build transactions in from the start. And how your transactions look like determines the rest of the system - performance, ergonomics, query semantics, everything needs to work nicely with your transaction subsystem.

So it’s only fair to ask about transaction support in Glowdust.

Excellent question. The answer is that there isn’t any yet.

But here’s what it will look like once it lands.

1. Transactions should be invisible by default

Most systems try to make transactions as transparent as possible. Relational databases will create a transaction for you for every SQL statement and commit it at the end.

Honestly, I like that. The DBMS should take care of ACID precisely so the user shouldn’t have to, and transactions are part of that. Most users just want to write some data and have the system behave reasonably. This means start a transaction, do the write, commit. This is the by far most common path and it should be easy.

So, design note #1: The user should, by default, not have to worry about transactions.

I think Glowdust is well positioned to work like this. The standard unit of interaction is a Glowdust program, a sequence of reads, writes and queries that return results. It is not complicated to envision this as surrounded by an automatically started transaction that commits if the program executes successfully.

For example, this glowdust program that writes, reads and prints a value:

square(4) = 16;
a = square(4);
println(a);

can be automatically surrounded by a transaction, semantically equivalent to:

txId = beginTx();

square(4) = 16;
a = square(4);
println(a);

commitTx(txId);

I expect this to be the most common pattern of writes for Glowdust and it should be just that easy.

2. Transactions should be possible to access

That’s the next step in the visibility ladder. It should be possible to have control over a transaction from within, and have access to some data about it.

The most obvious example is explcit commit or rollback, but a similar argument can be made about accessing timestamps (start time, for example) or information about the state of the system, like which was the latest transaction committed when the current transaction started.

PostgrsQL, for example, allows this by having an implicit object available in the transaction, accessible through either standard SQL or bespoke functions

Functions are the only thing that Glowdust does, so having some bespoke functions that return a transaction identifier and information about it is also very simple:

txId = current_tx();

start_time = tx_start_time(txId);
return start_time;

commitTx(txId);

Again, this API doesn’t exist, but it would be really easy to add, and it keeps transactions invisible unless the user wants to access them.

The important thing to note at this level is that the transaction, as an object, is not directly accessible. The txId is an identifier, not the transaction itself. There are no methods on txId, it’s just an opaque Glowdust struct.

Which brings us to design note #2: Transactions should be accessible through identifiers

But why identifiers and not an object? Let’s look at the third level of visibility:

3. Transactions should be unbound from implicit contexts

In the previous two points, transactions have lived in some undefined context that is at least accessible from the query as it executes. But what is that context, exactly?

Most systems (including those that conform to JDBC) use the network connection. That mostly makes sense - for OLTP systems, if the connection is disrupted, the transaction should roll back. This means the commit() and rollback() calls are on the Connection object (at least at the lower driver levels) and “autostart/autocommit” behavior is also controlled on the connection level.

But I think we can do a bit better, without complicating things. I want to be able to resume a transaction from a different connection.

The reason is job control and long running transactions.

Say I start an import from a remote source, that I expect to take a while. I don’t want to keep the connection open while the import is ongoing, but I definitely want it to execute as a transaction. Where does this long running, detached transaction live? Surely not on the network connection of the client that started it. That may go away. The user may issue the command and close their laptop and move, then after a while connect again to see how their import is progressing, maybe do a query in that context or roll it back.

We could take the example of the screen CLI utility, where we can attach to an ongoing session by id.

See where I am going with this?

txId = beginTx();

// start long running import

return txId;

// here you get the txId string, save it in a variable
// Close connection
// ...go for a walk
// return and open your Glowdust client

attachTx(txId);

// do a match, get counts or rollback

Even better, there is no additional mental overhead for the user with this functionality. They can completely ignore the serializability of the txId and just stick with commit() and rollback() from within the query and be done.

But making it possible to have different physical clients (even if they are the same logical one) connect and resume a transaction seems like the proper abstraction for job control in an OLTP system.

In other words, obvious design note #3: Transactions should be independent, identifiable contexts.

I still have a couple of questions to answer.

The first is when does a transaction start?

beginTx() vs attachTx()

The way I see it, a Glowdust program should determine if it needs a new transaction to start or if it should resume an existing transaction.

Starting a transaction happens with a beginTx() call, that much is easy. The identifier that returns can then be passed as metadata to other programs which can call (implicitly or explicitly) an attachTx() call to execute in the same transaction.

If a program is submitted without a txId metadata and doesn’t start with a beginTx() call, then it should have one executed for it and commit() at the end. The user can still extract the txId and pass it around, but I don’t think they should be able to do much with it as it should always be committed at the end.

This lets drivers, for example, to have an explicit Transaction object on the client side that can be managed independently of the connection, if they wish, but still retain its state in sync with the server because of the common identifier.

Concurrent transaction access from multiple clients

The second question is what about two clients connecting concurrently at the same transaction?

There is nothing in the API so far disallowing this. Actually, this could be something the system can explicitly allow.

I think that would be a bad idea.

I don’t want transactions to be thread safe. They should be single threaded objects, period.

So I think that, to start with, if a client tries to attach to a transaction that already has a client attached, that will result in an error. That doesn’t seem hard to implement.

This lets transactions single threaded and it leaves open the field for interesting LSM Tree shenanighans to make WAL obsolete and give MVCC a decent chance to work over SSDs.

But we’re getting into implementation territory now, and I’ll leave this for another post.

As always, let me know your thoughts on Mastodon

And, if you find this interesting enough, you may want to donate towards my costs as an independent developer.