Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
Cacheing class for Python - using a local SQLite database as a key/value store

Question from Tom: What did you have for lunch yesterday?
Answer: A Sandwich, a piece of cake and some fruit.

Question from Dick: What did you have for lunch yesterday?
Answer: A Sandwich, a piece of cake and some fruit.

Question from Harry: What did you have for lunch yesterday?
Answer: A Sandwich, a piece of cake and some fruit.

The answer's the same each time, no matter who asks, and it will remain the same for 24 hours no matter how many times is asked. And if working out the answer is quite a bit of work / involves getting someone else to remind you, then wouldn't it be better so remember the result and just replay it? That's the principle of caching an answer.

On yesterday's course, we were collecting a JSON object from a remote server, and doing so time and time again. That was inefficient use of the remote server, relatively slow, and an excellent demonstration case for caching the data returned which was originally called up for Tom, but is equally valid for Dick, Harry, Fiona, Betty, Susan and Bob. As a "control" case, the JSON loader example is [here] and I wrote about it [here].

Principle of basic cacheing:
• ask if we already have the data locally
• if YES, supply the local copy
• if NO, read from the remote source and store a copy locally

I've added code (in Python) that caches our JSON object into a local file - [here] - into the control example. There were a few bells and whistles required:

a) We had to see if the cached file actually existed (we checked how old it was) before we read it:

    cachetime = os.path.getmtime("cachedrecord.txt")
    cachetime = 0

b) We had to check to see if we had data that was out of date (in the example, data may live for just 20 seconds)

  if time.time() - cachetime > 20:

c) We had to handle the case of being unable to open the file to store the cached data:

    fho = open("cachedrecord.txt","w")

But there are two things that could be improved in that code:
• We're only caching a single specific value and
• We've added the caching code within our main program class; it's a useful utility and should be its own class for reuse and support purposes!
So during the Intermediate Python Course we improved on it!

The refactored application is [here] ... and the cache code has been reduced to:

  import kvcache
  localstore = kvcache.kvcache("http://www.wellho.net/services/*.json")
  stuff = localstore.get("pix")

Yes - that's all I need in my application! Of course, all the hard work's done within the class, where I used an SQLite3 database to store and retreive multiple records based on a key - "pix" in my example, but the system can store and pull back based on any unique test string key. The source code of the kvcache class is [here].

There are just two methods on the class:
• a constructor into which I pass the base URL (with a "*" for where the key is to be substituted in), and optionally a time to live (default 20 seconds) and a database file name for the class to use (defauly /tmp/mycache)
• a get method, which returns the record requested; pass in the key that you're looking for (substitute for the "*" in the URL in the constructor) and it will do the rest ... from setting up the database table, to pulling the record back, storing it based on key, checking timestamps, deleting the old record if it's expired.
All in all, this is a "classic" demonstration of the use of a class; the complicated stuff is encapsulated in the class, and all the user has to do is make a couple of simple calls ...

We could refactor further - I could replace the storage engine in the class with memcached for example, allowing cached responses to be held in memory and shared between a number of servers. And the beauty of the OO approach is that all my applications that use the kvcache should work without alteration once my refactoring has been completed with the same API.

(written 2013-05-14, updated 2013-05-18)

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
Y113 - Python and SQL databases
  [4537] example of SQLite using a local database file through SQLalchemy - (2015-10-14)
  [4535] SQLAlchemy - first examples with a Python Object Relationship Mapping system - (2015-10-14)
  [4445] Graphing presentations in Python - huge data, numpy and matplotlib - (2015-02-28)
  [4436] Accessing a MySQL database from Python with mysql.connector - (2015-02-21)
  [4024] SQL databases from Python - an SQLite example - (2013-03-02)
  [3139] Steering our Python courses towards wxPython, SQLite and Django - (2011-01-19)
  [3136] A framework with python - Django - first steps - (2011-01-17)
  [2790] Joining a MySQL table from within a Python program - (2010-06-02)
  [2786] Factory methods and SqLite in use in a Python teaching example - (2010-05-29)
  [2746] Model - View - Controller demo, Sqlite - Python 3 - Qt4 - (2010-04-29)
  [2745] Connecting Python to sqlite and MySQL databases - (2010-04-28)

Y117 - Python - Already written modules
  [4710] Searching a Json or XML structure for a specific key / value pair in Python - (2016-10-30)
  [4708] Scons - a build system in Python - building hello world - (2016-10-29)
  [4697] Month, Day, Year number to day of week and month names in Python - English and Swedish - (2016-06-23)
  [4696] Programming with random numbers - yet re-using the same values for testing - (2016-06-22)
  [4452] Binary data handling - Python and Perl - (2015-03-09)
  [4441] Reading command line parameters in Python - (2015-02-23)
  [4085] JSON from Python - first principles, easy example - (2013-05-13)
  [3479] Practical Extraction and Reporting - using Python and Extreme Programming - (2011-10-14)
  [3465] How can I do an FTP transfer in Python? - (2011-10-05)
  [3442] A demonstration of how many Python facilities work together - (2011-09-16)
  [2931] Syncronise - software, trains, and buses. Please! - (2010-08-22)
  [2890] Dates and times in Python - (2010-07-27)
  [2506] Good example of recursion in Python - analyse an RSS feed - (2009-11-18)
  [2020] Learning Python - many new example programs - (2009-01-31)

Back to
JSON from Python - first principles, easy example
Previous and next
Horse's mouth home
Forward to
Python network programming - new FTP and socket level examples
Some other Articles
Test Driven Development in Python - Customer Comes First
Quick and easy - showing Python data hander output via a browser
Some tips and techniques for huge data handling in Python
Python network programming - new FTP and socket level examples
Cacheing class for Python - using a local SQLite database as a key/value store
New Pictures - Melksham Pack Horse Bridge
Spring at Well House Manor - Teas and Coffees, Museum, Garden, Rooms
Training around the world - easy payment in pounds Sterling
Pushing down the advertised price, pushing up the total price charged.
4734 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2017: 404 The Spa • Melksham, Wiltshire • United Kingdom • SN12 6QL
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4086_Cac ... store.html • PAGE BUILT: Sat May 27 16:49:10 2017 • BUILD SYSTEM: WomanWithCat