libattachsql, the next-generation c connector for mysql

libAttachSQLThe Next-Generation C Connector for MySQL

by Andrew Hutchings (LinuxJedi), HP Advanced Technology Group

About LinuxJedi

• Worked for Sun/Oracle, MySQL Support Engineer• Worked at Rackspace, Drizzle Software Engineer• Worked at SkySQL, Senior Sustaining Engineer• Managed two teams, several projects at HP Cloud• Co-Author of MySQL 5.1 Plugins Development

About HP's Advanced Techonology Group

• An Open Source think-tank• Fosters collaboration throughout HP and Open Source communities

Other MySQL C Connectors

• libmysqlclient (GPLv2/proprietary)• libdrizzle (3-clause BSD)• Maria DB Client (LGPLv2.1 (sort-of))

None of these have an official fully non-blocking API!

Introducing libAttachSQL

libAttachSQL 1.0 Features

• Non-blocking API• Apache 2.0 license• Very lightweight (1.0.0 shared binary is 501KiB)• Can use OpenSSL and zlib for encryption and compression• Server side and (sort-of) local prepared statements• Good documentation from the start (http://docs.libattatchsql.org/)

Compatible With

• Ubuntu 12.04 and up• RedHat/CentOS 6.x and up• Windows (via. MinGW cross-compile)• GCC and CLang

Why not libdrizzle?

• libdrizzle 5.1 has a blocking API• Internal state loop is ugly to deal with• Networking code was hard to work with

Good things about libdrizzle

• Lightweight• Nice, simple API• Well tested (Jenkins CI)• Easy to contribute to• Liberal license

Why libAttachSQL?

• Modern techniques• All of the good points of libdrizzle, none of the legacy• Designed from the ground-up for use with MySQL

Event loops are web scale!

Non-blocking Internals

• Using libuv - a library born out of Node.js• Uses epoll/poll/kqueue in *nix• Uses I/O Completion Ports in Windows

Very Open Development

• GitHub (with issues & pull requests)• GitHub Pages + Pelican• Travis CI• Waffle.io• Read The Docs• Coverity Scan• IRC

Travis Tests

• GCC 32bit & 64bit• CLang 3.0 32bit & 64bit• CLang 3.4 64bit• No OpenSSL & zlib• Valgrind• Cppcheck• CLang Scan-Build• Documentation (Sphinx nit-pick mode)

Software Using libAttachSQL

• Sysbench• AttachBench

Basic Query Example

con= attachsql_connect_create("localhost", 3306, "test", "test", "testdb", NULL);attachsql_query(con, strlen(query), query, 0, NULL, &error);

Things to Note

• We haven't actually even connected yet!• No query/result object

while ((ret != ATTACHSQL_RETURN_EOF) && (error == NULL)){ ret= attachsql_connect_poll(con, &error); if (ret != ATTACHSQL_RETURN_ROW_READY) { continue; }

row= attachsql_query_row_get(con, &error); columns= attachsql_query_column_count(con); ... attachsql_query_row_next(con);}

Escaped Queries

const char *query= "SELECT * FROM t1 WHERE name = ? AND age > ?";attachsql_query_parameter_st param[2];

const char *name= "fred";uint32_t age= 30;param[0].type= ATTACHSQL_ESCAPE_TYPE_CHAR;param[0].data= (char*)name;param[0].length= strlen(name);param[1].type= ATTACHSQL_ESCAPE_TYPE_INT;param[1].data= &age;param[1].is_unsigned= true;attachsql_query(con, strlen(query), query, 2, param, &error);

Prepared Statements

const char *query= "SELECT * FROM t1 WHERE name = ? AND age > ?";

attachsql_statement_prepare(con, strlen(query), query, &error);while((ret != ATTACHSQL_RETURN_EOF) && (error == NULL)){ ret= attachsql_connect_poll(con, &error);}

const char *name= "fred";uint32_t age= 30;attachsql_statement_set_string(con, 0, strlen(name), name, NULL);attachsql_statement_set_int(con, 1, age, NULL);attachsql_statement_execute(con, &error);ret= ATTACHSQL_RETURN_NONE;while ((ret != ATTACHSQL_RETURN_EOF) && (error == NULL)){

ret= attachsql_connect_poll(con, &error); if (ret != ATTACHSQL_RETURN_ROW_READY) { continue; } attachsql_statement_row_get(con, &error); printf("ID: %d, ", attachsql_statement_get_int(con, 0, &error)); size_t len; char *name_data= attachsql_statement_get_char(con, 1, &len, &error); printf("Name: %.*s, ", (int)len, name_data); printf("Age: %d\n", attachsql_statement_get_int(con, 2, &error)); attachsql_statement_row_next(con);}

Connection Groups

• Group many connections into a single event loop• Many queries can run simultaneously on a single thread• Uses a callback API when there events to process• Can't be used for Prepared Statements (yet)

Connection Groups Example

group= attachsql_group_create(NULL);con[0]= attachsql_connect_create("localhost", 3306, "test", "test", "testdb", NULL);attachsql_group_add_connection(group, con[0], &error);attachsql_connect_set_callback(con[0], callbk, &con_no[0]);

attachsql_query(con[0], strlen(query1), query1, 0, NULL, &error);

while(done_count < 3){ attachsql_group_run(group);}attachsql_group_destroy(group);

void callbk(attachsql_connect_t *current_con, attachsql_events_t events, void *context, attachsql_error_t *error){ ... switch(events) {

case ATTACHSQL_EVENT_CONNECTED: printf("Connected event on con %d\n", *con_no); break;

case ATTACHSQL_EVENT_ERROR: printf("Error exists on con %d: %d\n", *con_no, attachsql_error_code(error)); attachsql_error_free(error); break;

case ATTACHSQL_EVENT_EOF: printf("Connection %d finished\n", *con_no); done_count++; attachsql_query_close(current_con); break;

case ATTACHSQL_EVENT_ROW_READY: row= attachsql_query_row_get(current_con, &error); ... attachsql_query_row_next(current_con); break;

Error Handling

attachsql_error_t *error= NULL;ret= attachsql_connect_poll(con, &error);if (error != NULL){ printf("Error occurred: %s", attachsql_error_message(error)); attachsql_error_free(error);}

Project Status

• 4 months in development (coding started 4th July)• 3 alpha releases, 2 beta releases, 1 RC release• First GA (1.0.0) is out now!• Packages for Ubuntu and RHEL/CentOS

Future

• Python wrapper• Other language wrappers• Server API?• Tools built around libAttachSQL

Questions?

Thank you

libAttachSQL - http://libattachsql.org/Freenode IRC - #libAttachSQLGitHub - http://github.com/libattachsql/libattachsql

Email andrew@linuxjedi.co.ukTwitter @LinuxJedi or @libAttachSQL