Tuesday, August 11, 2015

Cassandra Table doesn't exist but can't create it either

Hi All,

While starting to build a new data model in Cassandra i've created a couple of tables with a certain schema,
and of course like anything you create in software development, only one thing is certain - It will change!

So i had the need to alter / drop the table (column family), and recreate it with a different schema.



A quick versions and environment background: (Because most of the times it really matters).
I'm running Cassandra cluster in version 2.1.7, with the Datastax Community AMI, on AWS.

While dropping the table everything looked fine, it even disappeared from the listing.
But when i came to creating it again, it did nothing,

I was running a "CREATE" clause similar to this:

CREATE TABLE IF NOT EXISTS some_keyspace.some_table (
    "field" timestamp,
    "another_field" text,
    "third_field" int,
    "metadata" map<text, text>,
    PRIMARY KEY(("field", "another_field"), "third_field")
);

After it did nothing, I removed the "IF NOT EXISTS" from the statement, and got an error the the table already exists.
That was really strange, so if the table exists, you'd probably say:
"Let's delete it! :)",
so i went on doing so... with the statement:


DROP TABLE some_keyspace.some_table;

And got the next strange error:

(com.datastax.driver.core.exceptions.DriverException: Host replied with server error: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 738a4cb0-3e99-11e5-b4e4-bfa8a942090e; expected 6c43a8c0-3e99-11e5-9591-a394170523a0)))

(I left out all of the IP addresses and server names, but the bottom line is that i wasn't able to neither Create nor Drop the wanted table.)

Seem like the UID of the table i was trying to drop was not existent, and nothing was happening.

After trying to refresh the status by any means via the "nodetool", nothing helped.
In the beginning i thought that it was a corruption of the data, and tried to check in the "system" column family, if there were any traces of the old table ("some_table"), but it couldn't find any.

So, what do you do when nothing works?? Yes..."Restart it".

The solution eventually after trying almost everything, was to gracefully restart the cassandra cluster, node by node.

For each node I did the following steps:
1) I used the drain operation: "nodetool drain".
(For the node to stop accepting requests and be ready for shutdown - you can read more about it here)
2) "sudo service cassandra restart".
Because we are running the cassandra as a service in all the nodes.
3) I checked out the log at "/var/log/cassandra/system.log" to see that all was fine and restored itself to a running state with no errors.
4) Once finished I went on to the next node.

I've tried to recreate the problem,
Created a new table, inserted some data into it, dropped it,
And all seems to work fine.

So if any of you had the same problem, I would love to hear two things from you:
1) How did you solve it? 
2) More important: How to recreate the problem :)

Hope this helps someone in the future!

No comments:

Post a Comment