Cannot use utf8mb4 character set with CloudSQL on AppEngine Python

Question

I installed an instance of CloudSQL that I am trying to use with my Django application in AppEngine. I confirmed that the server is configured to use the utf8mb4 symbol set using the CloudSQL console for my database:

 utf8mb4 utf8mb4_unicode_ci 

If I connect directly to mysql cli, I can successfully insert and read emojis. However, if I insert the same emoji characters through the Django admin, it is simply pasted as "???".

I tried to ensure that the MySQLdb-python client uses utf8mb4 with:

 'ENGINE': 'django.db.backends.mysql', ... 'OPTIONS': { 'charset': "utf8mb4", } 

But this makes me get the following error in AppEngine:

 (2019, "Can't initialize character set utf8mb4 (path: /usr/local/mysql/share/charsets/)") 

My app.yaml uses the "latest" MySQLdb library:

 libraries: - name: MySQLdb version: "latest" 

Answer 1

I just chatted with Google and got everything that works for our instance!

The standard way to get utf8mb4 to work in Django is to specify it as DATABASES ['default'] ['OPTIONS'] in settings.py, for example:

 'OPTIONS': {'charset': 'utf8mb4'}, 

This raises an OperationalError in the App Engine, on MySQLdb 1.2.4b4 / 1.2.4 / 1.2.5; which obviously means that the utf8mb4 character set is missing in the MySQL C compiler to compile Google C.

Uncheck this option.

The workaround is to manually call SET NAMES; edit lib / django / db / backends / mysql / base.py and add the line conn.query ("SET NAMES utf8mb4") to DatabaseWrapper.get_new_connection so that it looks like this:

 def get_new_connection(self, conn_params): conn = Database.connect(**conn_params) conn.encoders[SafeText] = conn.encoders[six.text_type] conn.encoders[SafeBytes] = conn.encoders[bytes] conn.query("SET NAMES utf8mb4") return conn 

Make sure utf8mb4 is also included in the backend. The migration commands in the Django App Engine tutorial lead to the Cloud SQL instance configured for utf8. I needed to run these commands in order to include utf8mb4 in two tables:

 ALTER TABLE polls_question CONVERT TO CHARACTER SET utf8mb4; ALTER TABLE polls_choice CONVERT TO CHARACTER SET utf8mb4;