Problem: I need a bit of data available to all templates
I decided to add a sidebar that has a list of posts - a standard "Archive" bit with links. So, naturally, I set about modifying all my views to include a helper function that returns data. I went through each of them and had them return the output of a function into the context. And hey, it worked!
But, there's an issue with how this was written. Django encourages a DRY philosophy - Don't Repeat Yourself. And I was repeating myself quite a bit here. Going through each view and doing the same thing is inefficient and can get to be hard to maintain in large codebases. This blog won't ever be all that big, but, I figured I'd try to find a way to make things better. You know, do it right. So I hit Google and found a bunch of possible answers. One, though, seems like the "right" way to do it.
Solution: Context Processors
Context processors in Django work to gather and process data at runtime. From there, you can have them return data and make it available to templates. There appears to be additional functionality to context processors, and I'll have to look into that in the future. But for now, this seems to do exactly what I want.
First, I created a file in my blog module (since this is a blog-related function) called 'context_processors.py'. I created my function, which had to return an ordered dict of years containing a list of post titles and url slugs.
posts = Posts.objects.filter().values('title','posttime','slug').order_by('-posttime')
archive = {}
for post in posts:
try:
year = archive.get(str(post['posttime'].year), [])
year.append(post)
archive[str(post['posttime'].year)]=year
except:
pass
archive = sorted(archive.items())
return {'sidebar': archive}
Quick and easy. It looks like Django's context processing loads this dict into a high level of the templating system, making the variable "sidebar" available to templates.
*(Todo: Add try-catch for getting data, I haven't experimented with what happens when a context processor has an exception yet, and if there are no posts, this won't return anything.)
After that, I had to add it to settings.py:
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [ os.path.join(BASE_DIR, 'templates'), ],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
'blog.context_processors.sidebar_archive_processor',
],
},
},
]
So that loads sidebar_archive_processor from blog.context_processors. Very straightforward. From here, it's just a simple matter of accessing that data in the template.
Here's the loop I used:
{% for year,posts in sidebar reversed %}
<div class="sidebar-box">
{{ year }}
<div id="sidebar-{{ year }}">
{% for post in posts %}
<a href="{% url 'blog_slug' post.slug %}"><div class="sidebar-link sidebar-button">{{ post.title }}</div></a>
{% endfor %}
</div>
</div>
{% endfor %}
I wanted to have it so that the most recent posts appeared first, so I loop through the sidebar backwards. I've given each sidebar div a unique id so we can eventually hook javascript into it for an accordion-style collapse. Oh, and I could have done a regroup here, but I like to keep code out of the template as much as possible, so I figured it would be best to put that logic in the context processor itself.
And there you have it. Works as intended, feature addition complete. I'll have to look into the other things that context processors can do next. Or, maybe start adding other modules to the site. Right now the Games, Code, and About links just go link to blog_home, so that should probably be updated. Those placeholders should probably have someplace more apt to link to.
Now that wasn't too hard. Imported the old db into the database server, created a script that connects to that database using pymysql, loops through the old posts and returns a data structure. Use some Django model bits to import the data I needed... ah well, might as well just show you:
# blog/oldblog.py
from blog.models import Posts
import datetime
import pymysql, pymysql.cursors
usermap = {
'2': '1',
'3': '2',
}
def runimport():
connection = pymysql.connect(settings)
try:
with connection.cursor() as cursor:
sql = "select ownerid,date,post,title from blog order by date desc"
cursor.execute(sql)
result = cursor.fetchall()
finally:
connection.close()
for record in result:
record['date'] = datetime.datetime.utcfromtimestamp(float(record['date']))
importrecord(record['title'],record['post'],usermap[str(record['ownerid'])],record['date'])
def importrecord(title,content,author,date):
try:
p = Posts(title=title, content=content, author_id=author)
p.save()
p.posttime = date
p.save()
print("Post import successful: {}".format(title))
except:
print("--Error: Could not import post: '{}'".format(title))
Then, from there, it's into the Django shell using the production server settings:
>>> from blog.oldblog import runimport,importrecord
>>> runimport()
Code's a little jank and some of the metadata that was part of the old blog system was lost, but that's okay. I can take the hit. Also, a few records failed to import due to not having titles. No big loss there, they were from when I was first building the old blog from scratch in php. Almost entirely "test post", "test post 2", "asdf" type posts. I had to convert the unix timestamp I was using in the old blog (bad) to a proper datetime for the new one (good); Super straightforward with datetime.
Now, this solution worked because I don't have a large dataset. If the blog was millions of lines of posts, I definitely wouldn't want to return the entire table in a single variable; I'd want to return one record at a time, and move the cursor down the dataset, allowing the script to free up memory as it moved along.
There's also the issue of the static usermap. I did it this way because:
- It was quick and dirty
- I only had 2 users to import posts from
The right way to do it would be to do a nested sql lookup against the 'users' table in the old blog, and map the old user to the new user that way. But I didn't maintain usernames across those databases; Old usernames are our handles, new usernames are our names. We'd need a static map anyway.
Last issue, no meaningful error handling. The old posts that failed to import don't get handled, just ignored with a message. But, for reasons stated above, no big deal for something this quick.
Overall: Success. Quick and dirty works a lot of the time, especially for one-off import scripts like this one. Didn't take long to hammer out all the kinks, like the date change thing you may have noticed. Thing about the Django DateTimeField with auto_now_add=True, is even if you specify the date in the field initially, it'll get set to the current time when the record is added. So you've gotta save it, update the time, and then save it again. I'm not sure if there's a way around that, but this method seems to work.
btw, the new blog system makes posting longer/more technical posts much easier and more fun. So I'll probably be doing a lot more of them in the future.
Peace
Trying out this whole Django thing, and so far I'm really digging it. Modules (called "apps" in django) make adding features without affecting other code pretty straightforward. The database models make just about everything a lot easier and faster to build. Even gunicorn didn't give me any trouble, and I thought it would. Sweet.
So, here we are, a more minimalist approach to Drunkserver. More to come as we update the backend and frontend.