This data as json
| sha | message | author_date | committer_date | raw_author | raw_committer | repo | author | committer |
|---|---|---|---|---|---|---|---|---|
| 31b21f5c5e15fc3acab7fabb170c1da71dc3c98c | Moved all SQLite queries to threads SQLite operations are blocking, but we're running everything in Sanic, an asyncio web framework, so blocking operations are bad - a long-running DB operation could hold up the entire server. Instead, I've moved all SQLite operations into threads. These are managed by a concurrent.futures ThreadPoolExecutor. This means I can run up to X queries in parallel, and I can continue to queue up additional incoming HTTP traffic while the threadpool is busy. Each thread is responsible for managing its own SQLite connections - one per database. These are cached in a threadlocal. Since we are working with immutable, read-only SQLite databases it should be safe to share SQLite objects across threads. On this assumption I'm using the check_same_thread=False option. Opening a database connection looks like this: conn = sqlite3.connect( 'file:filename.db?immutable=1', uri=True, check_same_thread=False, ) The following articles were helpful in figuring this out: * https://pymotw.com/3/asyncio/executors.html * https://marlinux.wordpress.com/2017/05/19/python-3-6-asyncio-sqlalchemy/ Closes #45. Refs #38. | 2017-11-05T02:21:44Z | 2017-11-05T02:21:44Z | 2946d096d0cdefdc017559e6b57e87658736e843 | 2946d096d0cdefdc017559e6b57e87658736e843 | 107914493 | 9599 | 9599 |