Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Cacheing class for Python - using a local SQLite database as a key/value store

Question from Tom: What did you have for lunch yesterday?
Answer: A Sandwich, a piece of cake and some fruit.

Question from Dick: What did you have for lunch yesterday?
Answer: A Sandwich, a piece of cake and some fruit.

Question from Harry: What did you have for lunch yesterday?
Answer: A Sandwich, a piece of cake and some fruit.

The answer's the same each time, no matter who asks, and it will remain the same for 24 hours no matter how many times is asked. And if working out the answer is quite a bit of work / involves getting someone else to remind you, then wouldn't it be better so remember the result and just replay it? That's the principle of caching an answer.

On yesterday's course, we were collecting a JSON object from a remote server, and doing so time and time again. That was inefficient use of the remote server, relatively slow, and an excellent demonstration case for caching the data returned which was originally called up for Tom, but is equally valid for Dick, Harry, Fiona, Betty, Susan and Bob. As a "control" case, the JSON loader example is [here] and I wrote about it [here].

Principle of basic cacheing:
• ask if we already have the data locally
• if YES, supply the local copy
• if NO, read from the remote source and store a copy locally

I've added code (in Python) that caches our JSON object into a local file - [here] - into the control example. There were a few bells and whistles required:

a) We had to see if the cached file actually existed (we checked how old it was) before we read it:

    cachetime = os.path.getmtime("cachedrecord.txt")
    cachetime = 0

b) We had to check to see if we had data that was out of date (in the example, data may live for just 20 seconds)

  if time.time() - cachetime > 20:

c) We had to handle the case of being unable to open the file to store the cached data:

    fho = open("cachedrecord.txt","w")

But there are two things that could be improved in that code:
• We're only caching a single specific value and
• We've added the caching code within our main program class; it's a useful utility and should be its own class for reuse and support purposes!
So during the Intermediate Python Course we improved on it!

The refactored application is [here] ... and the cache code has been reduced to:

  import kvcache
  localstore = kvcache.kvcache("http://www.wellho.net/services/*.json")
  stuff = localstore.get("pix")

Yes - that's all I need in my application! Of course, all the hard work's done within the class, where I used an SQLite3 database to store and retreive multiple records based on a key - "pix" in my example, but the system can store and pull back based on any unique test string key. The source code of the kvcache class is [here].

There are just two methods on the class:
• a constructor into which I pass the base URL (with a "*" for where the key is to be substituted in), and optionally a time to live (default 20 seconds) and a database file name for the class to use (defauly /tmp/mycache)
• a get method, which returns the record requested; pass in the key that you're looking for (substitute for the "*" in the URL in the constructor) and it will do the rest ... from setting up the database table, to pulling the record back, storing it based on key, checking timestamps, deleting the old record if it's expired.
All in all, this is a "classic" demonstration of the use of a class; the complicated stuff is encapsulated in the class, and all the user has to do is make a couple of simple calls ...

We could refactor further - I could replace the storage engine in the class with memcached for example, allowing cached responses to be held in memory and shared between a number of servers. And the beauty of the OO approach is that all my applications that use the kvcache should work without alteration once my refactoring has been completed with the same API.

(written 2013-05-14, updated 2013-05-18)

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
Y117 - Python - Already written modules
  [2020] Learning Python - many new example programs - (2009-01-31)
  [2506] Good example of recursion in Python - analyse an RSS feed - (2009-11-18)
  [2890] Dates and times in Python - (2010-07-27)
  [2931] Syncronise - software, trains, and buses. Please! - (2010-08-22)
  [3442] A demonstration of how many Python facilities work together - (2011-09-16)
  [3465] How can I do an FTP transfer in Python? - (2011-10-05)
  [3479] Practical Extraction and Reporting - using Python and Extreme Programming - (2011-10-14)
  [4085] JSON from Python - first principles, easy example - (2013-05-13)
  [4441] Reading command line parameters in Python - (2015-02-23)
  [4452] Binary data handling - Python and Perl - (2015-03-09)
  [4696] Programming with random numbers - yet re-using the same values for testing - (2016-06-22)
  [4697] Month, Day, Year number to day of week and month names in Python - English and Swedish - (2016-06-23)
  [4708] Scons - a build system in Python - building hello world - (2016-10-29)
  [4710] Searching a Json or XML structure for a specific key / value pair in Python - (2016-10-30)

Y113 - Python and SQL databases
  [2745] Connecting Python to sqlite and MySQL databases - (2010-04-28)
  [2746] Model - View - Controller demo, Sqlite - Python 3 - Qt4 - (2010-04-29)
  [2786] Factory methods and SqLite in use in a Python teaching example - (2010-05-29)
  [2790] Joining a MySQL table from within a Python program - (2010-06-02)
  [3136] A framework with python - Django - first steps - (2011-01-17)
  [3139] Steering our Python courses towards wxPython, SQLite and Django - (2011-01-19)
  [4024] SQL databases from Python - an SQLite example - (2013-03-02)
  [4436] Accessing a MySQL database from Python with mysql.connector - (2015-02-21)
  [4445] Graphing presentations in Python - huge data, numpy and matplotlib - (2015-02-28)
  [4535] SQLAlchemy - first examples with a Python Object Relationship Mapping system - (2015-10-14)
  [4537] example of SQLite using a local database file through SQLalchemy - (2015-10-14)

Back to
JSON from Python - first principles, easy example
Previous and next
Horse's mouth home
Forward to
Python network programming - new FTP and socket level examples
Some other Articles
Test Driven Development in Python - Customer Comes First
Quick and easy - showing Python data hander output via a browser
Some tips and techniques for huge data handling in Python
Python network programming - new FTP and socket level examples
Cacheing class for Python - using a local SQLite database as a key/value store
New Pictures - Melksham Pack Horse Bridge
Spring at Well House Manor - Teas and Coffees, Museum, Garden, Rooms
Training around the world - easy payment in pounds Sterling
Pushing down the advertised price, pushing up the total price charged.
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4086_Cac ... store.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb