Simulating server-side cursors with MySQL Connector/Python

Update Feb 6 2014: I have edited the example code a bit. Note that this has never been included in any GA release of Connector/Python.

Last week, my colleague Massimo and I discussed how to handle big result sets coming from MySQL in Python. The problem is that MySQL doesn’t support server-side cursors, so you need to select everything and then read it. You can do it either buffered or not. MySQL Connector/Python defaults to non-buffered, meaning that you need to fetch all rows after issuing a SELECT statement. You can also turn on the buffering, mimicking what MySQL for Python (MySQLdb) does.

For big result sets, it’s better to limit your search. You can do this using an integer primary key or some temporal field for example. Or you can use the LIMIT keyword. The latter solution is what is used in the MySQLCursorServerSide cursor-class. Using the SELECT it creates a temporary table from which the fetch-methods will get the information. It is something people have probably implemented in their applications, but I hope this new class will make it easier since it’s done transparently.

The code is not pushed yet, but expect it to be available in next release. Here is an example how you could use it. This code selects cities staring with Z, loops over the result getting the country (yes, this is a simple join made difficult):

    cnx = db.connect(user='root', db='world')
    cur = cnx.cursor()
    curCity = cnx.cursor(db.cursor.MySQLCursorServerSide)

    curCity.execute(
        "SELECT ID,Name,CountryCode FROM City "
        "WHERE NAME LIKE 'Z%' ORDER BY ID")

    query = "SELECT Code, Name FROM Country WHERE CODE = %s"
    for city in curCity:
        cur.execute(query, (city[2],))
        country = cur.fetchone()
        print("{0} ({1})".format(city[1], country[1]))

    cur.close()
    cnx.close()

I guess the main advantage is that you can use two or more cursor objects with the same connection without the need of buffering everything in Python. On the MySQL side, the temporary table could go to disk when to big. It’s maybe slower, but keeping big result sets in memory ain’t good either.

Comments are welcome!

Multiple result sets in MySQL Connector/Python

Latest code of MySQL Connector/Python on launchpad has support for multiple result sets when you execute a stored procedure. We also changed the way the actual result of the routine is returned to conform to PEP249.

Here is some example code: it creates a stored procedure which generates 2 result sets. You can get the result by calling next_resultset(), which is returning a MySQLCursorBuffered.

    cur = cnx.cursor()

    cur.execute("DROP PROCEDURE IF EXISTS multi")
    proc = """
      CREATE PROCEDURE multi(IN pFac1 INT, IN pFac2 INT, OUT pProd INT)
      BEGIN
        SELECT 1,'a' as FooBar;
        SELECT 2;
        SET pProd := pFac1 * pFac2;
      END"""

    cur.execute(proc)
    result = cur.callproc("multi", (5, 6, 0))
    print "Result:", result
    
    extcur = cur.next_resultset()
    i = 1
    while extcur:
        rows = extcur.fetchall()
        print "Result set #%d:" % i
        print rows
        extcur = cur.next_resultset()
        i += 1
        
    cur.close()

The output:

Result: ('5', '6', 30)
Result set #1:
[(1, u'a')]
Result set #2:
[(2,)]

As mentioned above: this will be part of 0.1.4-devel release due next week. Comments are welcome through the associated bug report.

The wrapped jumbo prawns

The USA is not particulary famous for its cuisine. I must admit that it has been quite good the past 10 days. The Angus Burger at Hyatt, with mushrooms? Yummy!

Today I saw KFC advertising a new sandwish. All chicken, with cheese, with bacon in the middle. That’s right: bacon.

This evening I went for pasta at the restaurant. I started with chowder, which was great. Then the entree (maindish in US, go figure). Tagiatelle topped with prawns, jumbo ones. Ladies & gents, the sea creatures were wrapped in bacon.

It was good though.

Insert data into a VARCHAR field using NDB API: a solution

You are using MySQL Cluster and crazy enough to digest NDB API? Sick of SQL? Here’s a treat: a function to make C/C++ strings ready for inserting into a VARCHAR field. The special thing about them is that the length is prefixed in the first 2 bytes.

void make_ndb_varchar(char *buffer, char *str)
{
  int len = strlen(str);
  int hlen = (len > 255) ? 2 : 1;
  buffer[0] = len & 0xff;
  if( len > 255 )
    buffer[1] = (len / 256);
  strcpy(buffer+hlen, str);
}

Yes, you can use memcpy. Whatever floats your boat.

Lets use this function for a table t1, defined as follows (note: latin1!):

CREATE TABLE t1 (
  id INT UNSIGNED NOT NULL,
  vc VARCHAR(128),
  vclong VARCHAR(1280),
  PRIMARY KEY (id)
  ) ENGINE=NDB DEFAULT CHARSET=latin1

Here is part of the code, simplified for this post:

 char vc[128+1]; // Size of 'vc', +1 for length info
 char vclong[1280+2]; // Size of 'vclong', +2 for length info
 ..
 make_ndb_varchar(vc, "NDB API kicks ass");
 operation->setValue("vc", vc);
 ..

The above example uses latin1. You could use Unicode, but that would probably mean converting from one encoding to the other using iconv. That’s another story.

This post complements Johan Andersson’s blog entry. Thanks to my colleagues Mats and Roger who helped me with a silly problem today regarding this function.