SQLAlchemy: print the actual query

Question

I would really like to be able to print valid SQL for my application, including values, rather than binding parameters, but it is unclear how to do this in SQLAlchemy (by design, I'm sure).

Has anyone solved this problem in general?

Answer 1

This works in python 2 and 3 and is a little cleaner than before, but requires SA> = 1.0.

from sqlalchemy.engine.default import DefaultDialect from sqlalchemy.sql.sqltypes import String, DateTime, NullType # python2/3 compatible. PY3 = str is not bytes text = str if PY3 else unicode int_type = int if PY3 else (int, long) str_type = str if PY3 else (str, unicode) class StringLiteral(String): """Teach SA how to literalize various things.""" def literal_processor(self, dialect): super_processor = super(StringLiteral, self).literal_processor(dialect) def process(value): if isinstance(value, int_type): return text(value) if not isinstance(value, str_type): value = text(value) result = super_processor(value) if isinstance(result, bytes): result = result.decode(dialect.encoding) return result return process class LiteralDialect(DefaultDialect): colspecs = { # prevent various encoding explosions String: StringLiteral, # teach SA about how to literalize a datetime DateTime: StringLiteral, # don't format py2 long integers to NULL NullType: StringLiteral, } def literalquery(statement): """NOTE: This is entirely insecure. DO NOT execute the resulting strings.""" import sqlalchemy.orm if isinstance(statement, sqlalchemy.orm.Query): statement = statement.statement return statement.compile( dialect=LiteralDialect(), compile_kwargs={'literal_binds': True}, ).string 

Demo:

 # coding: UTF-8 from datetime import datetime from decimal import Decimal from literalquery import literalquery def test(): from sqlalchemy.sql import table, column, select mytable = table('mytable', column('mycol')) values = ( 5, u'snowman: ☃', b'UTF-8 snowman: \xe2\x98\x83', datetime.now(), Decimal('3.14159'), 10 ** 20, # a long integer ) statement = select([mytable]).where(mytable.c.mycol.in_(values)).limit(1) print(literalquery(statement)) if __name__ == '__main__': test() 

Gives this result: (tested on python 2.7 and 3.4)

 SELECT mytable.mycol FROM mytable WHERE mytable.mycol IN (5, 'snowman: ☃', 'UTF-8 snowman: ☃', '2015-06-24 18:09:29.042517', 3.14159, 100000000000000000000) LIMIT 1 

Answer 2

In the vast majority of cases, a “strict” SQLAlchemy statement or query is as simple as:

 print str(statement) 

This applies to both ORM Query and any select() or other expression.

Note : the following detailed answer is supported in sqlalchemy documentation .

To get an instruction compiled for a specific dialect or engine, if the operator itself is not yet tied to one, you can pass this to compile () :

 print statement.compile(someengine) 

or without engine:

 from sqlalchemy.dialects import postgresql print statement.compile(dialect=postgresql.dialect()) 

When providing an ORM Query object, to get the compile() method, we only need access to .statement first:

 statement = query.statement print statement.compile(someengine) 

regarding the initial caveat that related parameters should be “embedded” in the final string, the problem here is that SQLAlchemy usually does not ask this question, since it is correctly handled using Python DBAPI, not to mention circumventing related parameters are probably the most widely used security holes in modern web applications. SQLAlchemy has limited ability to perform this stringing under certain circumstances, such as when using DDL. To access this function, you can use the "literal_binds" flag passed to compile_kwargs :

 from sqlalchemy.sql import table, column, select t = table('t', column('x')) s = select([t]).where(tcx == 5) print s.compile(compile_kwargs={"literal_binds": True}) 

The above approach has reservations that it is only supported for basic types such as ints and strings, and in addition, if a bindparam without a predefined value is used directly, it will not be able to do this either.

To support inline literal rendering for types that are not supported, run a TypeDecorator for the target type, which includes TypeDecorator.process_literal_param :

 from sqlalchemy import TypeDecorator, Integer class MyFancyType(TypeDecorator): impl = Integer def process_literal_param(self, value, dialect): return "my_fancy_formatting(%s)" % value from sqlalchemy import Table, Column, MetaData tab = Table('mytable', MetaData(), Column('x', MyFancyType())) print( tab.select().where(tab.cx > 5).compile( compile_kwargs={"literal_binds": True}) ) 

output output as:

 SELECT mytable.x FROM mytable WHERE mytable.x > my_fancy_formatting(5) 

Answer 3

Given that what you want only makes sense when debugging, you can run SQLAlchemy with echo=True to log all SQL queries. For example:

 engine = create_engine( "mysql://scott:tiger@hostname/dbname", encoding="latin1", echo=True, ) 

This can also be changed for just one request:

echo=False - if True , Engine registers all operators, as well as repr() their parameter lists in the engine logger, by default sys.stdout . The echo Engine attribute can be changed at any time to enable or disable registration. If the string "debug" specified, the result lines will also be output to standard output. This flag ultimately controls the Python logger; see Configuring Logging for information on how to directly configure logging.

Source: SQLAlchemy Engine Configuration

If you use Flask, you can simply install

 app.config["SQLALCHEMY_ECHO"] = True 

to get the same behavior.

Answer 4

We can use the compilation method for this purpose. From the docs :

 from sqlalchemy.sql import text from sqlalchemy.dialects import postgresql stmt = text("SELECT * FROM users WHERE users.name BETWEEN :x AND :y") stmt = stmt.bindparams(x="m", y="z") print(stmt.compile(dialect=postgresql.dialect(),compile_kwargs={"literal_binds": True})) 

Result:

 SELECT * FROM users WHERE users.name BETWEEN 'm' AND 'z' 

Answer 5

So, let's build @zzzeek comments on @bukzor code. I came up with this to easily get a pretty-printed request:

 def prettyprintable(statement, dialect=None, reindent=True): """Generate an SQL expression string with bound parameters rendered inline for the given SQLAlchemy statement. The function can also receive a `sqlalchemy.orm.Query` object instead of statement. can WARNING: Should only be used for debugging. Inlining parameters is not safe when handling user created data. """ import sqlparse import sqlalchemy.orm if isinstance(statement, sqlalchemy.orm.Query): if dialect is None: dialect = statement.session.get_bind().dialect statement = statement.statement compiled = statement.compile(dialect=dialect, compile_kwargs={'literal_binds': True}) return sqlparse.format(str(compiled), reindent=reindent) 

I personally find it difficult to read sqlparse code, so I used sqlparse to reuse SQL. It can be installed using pip install sqlparse .

Answer 6

This code is based on the brilliant existing answer from @bukzor. I just added custom rendering for datetime.datetime type to Oracle TO_DATE() .

Feel free to update the code according to your database:

 import decimal import datetime def printquery(statement, bind=None): """ print a query, with values filled in for debugging purposes *only* for security, you should always separate queries from their values please also note that this function is quite slow """ import sqlalchemy.orm if isinstance(statement, sqlalchemy.orm.Query): if bind is None: bind = statement.session.get_bind( statement._mapper_zero_or_none() ) statement = statement.statement elif bind is None: bind = statement.bind dialect = bind.dialect compiler = statement._compiler(dialect) class LiteralCompiler(compiler.__class__): def visit_bindparam( self, bindparam, within_columns_clause=False, literal_binds=False, **kwargs ): return super(LiteralCompiler, self).render_literal_bindparam( bindparam, within_columns_clause=within_columns_clause, literal_binds=literal_binds, **kwargs ) def render_literal_value(self, value, type_): """Render the value of a bind parameter as a quoted literal. This is used for statement sections that do not accept bind paramters on the target driver/database. This should be implemented by subclasses using the quoting services of the DBAPI. """ if isinstance(value, basestring): value = value.replace("'", "''") return "'%s'" % value elif value is None: return "NULL" elif isinstance(value, (float, int, long)): return repr(value) elif isinstance(value, decimal.Decimal): return str(value) elif isinstance(value, datetime.datetime): return "TO_DATE('%s','YYYY-MM-DD HH24:MI:SS')" % value.strftime("%Y-%m-%d %H:%M:%S") else: raise NotImplementedError( "Don't know how to literal-quote value %r" % value) compiler = LiteralCompiler(dialect, statement) print compiler.process(statement) 

Answer 7

I would like to point out that the solutions above do not “just work” with non-trivial queries. One problem that I am facing is more complex types like pgsql ARRAY causing problems. I found a solution that just worked for me even with pgsql ARRAYs:

borrowed from: https://gist.github.com/gsakkis/4572159

The linked code seems to be based on an older version of SQLAlchemy. You will receive an error message indicating that the _mapper_zero_or_none attribute does not exist. Here's an updated version that will work with a newer version, you just replace _mapper_zero_or_none with bind. In addition, it supports pgsql arrays:

 # adapted from: # https://gist.github.com/gsakkis/4572159 from datetime import date, timedelta from datetime import datetime from sqlalchemy.orm import Query try: basestring except NameError: basestring = str def render_query(statement, dialect=None): """ Generate an SQL expression string with bound parameters rendered inline for the given SQLAlchemy statement. WARNING: This method of escaping is insecure, incomplete, and for debugging purposes only. Executing SQL statements with inline-rendered user values is extremely insecure. Based on http://stackoverflow.com/questions/5631078/sqlalchemy-print-the-actual-query """ if isinstance(statement, Query): if dialect is None: dialect = statement.session.bind.dialect statement = statement.statement elif dialect is None: dialect = statement.bind.dialect class LiteralCompiler(dialect.statement_compiler): def visit_bindparam(self, bindparam, within_columns_clause=False, literal_binds=False, **kwargs): return self.render_literal_value(bindparam.value, bindparam.type) def render_array_value(self, val, item_type): if isinstance(val, list): return "{%s}" % ",".join([self.render_array_value(x, item_type) for x in val]) return self.render_literal_value(val, item_type) def render_literal_value(self, value, type_): if isinstance(value, long): return str(value) elif isinstance(value, (basestring, date, datetime, timedelta)): return "'%s'" % str(value).replace("'", "''") elif isinstance(value, list): return "'{%s}'" % (",".join([self.render_array_value(x, type_.item_type) for x in value])) return super(LiteralCompiler, self).render_literal_value(value, type_) return LiteralCompiler(dialect, statement).process(statement) 

Tested up to two levels of nested arrays.