As we discussed in the previous blog we need to move away from classic ASP and into a modern web framework so that I can continue to evolve the site. Choosing a web framework often isn't easy since there are so many to choose from and I wouldn't want to get stuck on the wrong horse again.
I have been writing RESTful webservices in Python on the Flask microframework for a few years now and I think this would be a good fit for CC.
I have a lot of experience in working with Amazon Web Services and that is the platform that I will use for the the new CC website. This platform gives us a lot of freedom on how services are split up and allows for elastic scaling based on usage, giving a good balance between performance and cost.
The stack will sit on a Linux system with a Postgres database. The advantage of moving from windows/ms sql to linux/postgres is that there are no licensing fees and the operational costs will be slashed by half.
The problem is, however that there is very little common ground and I will be required to literally rewrite every single line of code. As we saw previously, this is well over 100 thousand lines (we're talking Harry Potter in VBScript here).
After mulling it over for a while and doing some tests I've concluded that the best course of action is to rewrite individual systems and run a new website alongside the old one for a while as I progress through the rewrite.
The first thing that needs to happen is to reduce the current website down to its fundimental parts and get rid of any features that are not used much. I want to do this first for a couple of reasons. Firstly, because these extra pieces of functionality are in my way when I'm breaking down the work that needs to be done and figuring out which part goes where, and also because I want to see if any of the features will be particularily missed by the members so that I'm not in the unfortunate position of having taken some toys away at the time I open the doors to the new site, which will cause bitterness when we should be celebrating.
Among the features that I want to remove are the following:
Spell checking
I hope no one is using this feature anymore since the spell check in current browsers is so much better.
Quick text
I personally use this for quick replies but I could just as well use a browser extension.
In-site bookmarks
This is something from the early days of CC and I don't see any reason to have this anymore. People can just use browser bookmarks.
Hangman and word morph games
They are not used much but maybe I will reimplement them or add some other games on the new site.
Private Forums
The way people communicate has changed. Currently these are hardly used at all except to discuss stories in private queues. Forum discussion of stories also isn't done much anymore but if needed I will implement a single story comment system instead.
World map
This is a fun little feature that I might re-implement later but not worth the effort now.
Reminders
People don't need reminders on CC, we have google calendar for that.
Statistics
The stats pages use a graphic library that I will likely not be able to migrate to the new server on aws. Probably at some point I will be able to make even better pages on the new site using some interactive library.
Non-inline crits
In the new site there will only be one critique type, which will be inline, possibly including in-paragraph commenting. To start off with I will remove the classic and template crit type options (the latter has already been removed) and new crits will always be inline. You will be able to submit a 'classic' crit by just leaving a comment in the 'comments before' box.
Other tools
I will probably move over the tools one by one towards the end of the development. I think the outlining workshop will probably be at the top of the list to get back in. The manuscript progress and word meter will also be high, but I'm not sure about the rest since most of the tools will require a lot of work and most of them don't get much use.
There are a few other little features that I will turn off soon and I know it will annoy some people, but believe me whan I say that it's for the greater good. Please let me know in the comments section though if you really miss some features and maybe they will make a comeback in the new CC somewhere down the line.
Forums
I am hoping to be able to use some off-the-shelf forum solution instead of rolling my own. This would save me roughly a quarter of the work overall, but it might cause forums not to be as tightly integrated into the rest of the site as they are today.
On the other hand, integrating modern forum software will give us much richer and nicer forum experience than we have today so I believe we will come out on top.
There are several instances where we have deep forum integration with other features, such as story comments and blog comments. I will need to cut these ties and make the forums a distinct application, separate of the core CC, but sharing authentication. The forums might even move to community.critiquecircle.com.
Because of this story comments and blog comments will likely not be able to utilize the forum system.
The Plan
We need to move from a monolithic classic ASP website written in VBScript to a modular web application written in Python utilizing cloud architecture. In short, how do we go from this:
to this:
The exact moving parts here are not very interesting to you guys but the important thing is that this is an entirely different architecture with a lot of things that need to be migrated or rewritten.
Although I would like to do it all at once I'm simply not able to. This will me a multi-month operation for sure and I will need to be in a position to allow both the new and old code to work at the same time.
To achieve this I will need to ensure that both sites are able to use the same database. AWS allows me to do this by using their hosted RDS service, taking the database out of the application box and making it available to several services. This is fundimental to the strategy.
Keep in mind through all of this that my immediate intention for this rewrite is not to improve the functionality of CC but just to get it done. We will look into improvements later, once we've transitioned 100% into a modern web framework. Therefore, after the bulk of the work is done, if we're lucky, CC will function much the same as it does today.
Phase 1 - Simplify the application
There is a lot of code in CC that has been built up over the years that isn't doing much and might be removed. There are also quite a few features that are not used much and will be time consuming to rewrite. I have already started removing unused tables, code and features and slimming down the database. This will make the move to AWS smoother. Sadly this means that some users will lose certain features that they like, at least temporarily.
Phase 2 - Move CC to AWS
Once CC is slim and thin I intend to move it as-is from the current bare-metal hosting provider to an AWS EC2 instance. I will be splitting up the DB and the application, but other than that there should be no change.
Once this is done we should have something like this:
I've already done some tests that look quite good. We should be able to get this part done in a reasonably short time, which will take care of the immediate problem of losing the current hardware.
Since the database will be on a dedicated machine managed by AWS I have hopes that performance will be more consistent than what we experience currently,
Phase 3 - Migrate to Postgres
Now we can start to migrate the database from Microsoft SQL Server to Postgres which is a much easier database engine to work with from the Python/Linux side (plus it's free). This work will involve moving the stored proceedures into code or postgres functions. There is luckily not a lot of high level functionality in the database but there is some. During this I will further simplify the database architecture and throw away unneeded functionality.
Since we will still be running the old vbscript website I will need to move the db functionality into VBScript even though it will be discarded soon. We also need to change some table structure to conform to postgres. This will be a reasonable amount of work but some perliminary tests indicate that this can be done and that classic vbscript can use postgres as a backend.
Again, once this is done CC should be running as it did before, but with a whole other engine under the hood. Now I will be all set to begin the actual transition into a new architecture.
Phase 4 - Bootstrap the new website
Now I can start to build the first 'hello world' application of the new website. This will involve setting up development and deployment pipelines, setting up docker, lambdas and redis stacks, choosing modules and web stack pieces, getting OpenID Connect up and running and getting everything to work smoothly on AWS. This will be completely behind the scenes and with minimal changes to CC itself.
Once this is done I will be able to start writing features against the common datastore. We will likely have two websites at this point, the old one in classic ASP and a new one in python. Some of the links on the website will lead to the old website and some to the new. The transition between the two will hopefully be rather seamless.
Phase 5 - Reimplement core functionality
Now I will start to build up a brand new website from scratch, maybe hosting it on develop.critiquecircle.com. This website will use the same backend database as the old site and should be usable by members. Once features start to stabilize I will probably encourage users to transition to the new site so that we can see how it behaves. However, when you enter many of the sections of the site such as stories, forums or tools you will jump to the old site since the features don't exist on the new one yet.
I have not decided in which order the features will be implemented, but login, settings, member info, blogs, admin functionality are probably high on the list.
At this point the architecture looks something like this (removing a bunch of things for simplicity)
Basically we have two independent websites using the same database with some features being delivered by the old one and others by the new one.
Phase 6 - Reimplement stories
This is the big one. In this phase I will rewrite stories, queues, novels and crits. There are so many features, nooks and crannies in these systems that will need to be worked on that I fear this will take a long time.
It's very possible that some features will not make it in on the first pass and from the user's perspective we will be regressing in terms of functionality. Hopefully by the time we're completely done the stories section will function much like it did before.
Phase 7 - Move forums into an off-the-shelf module
Now I will deprecate the forum functionality in favor of a forum package that I will hopefully not need to implement myself. The forums will probably look very different from that they are today and some features will certainly be lost. However, we should get much snappier and more modern forums as a consequence.
At the end of this phase only the tools section will remain on the old website and those will be moved into legacy.critiquecircle.com. I'm not certain how many of the tools will be moved into the new site at all, it will have to be seen.
Phase 8 - Address remaining functionality
Any remaining functionality will be moved into the new site or completely removed. Some features might then be reimplemented on the new site later on and others will disappear forever.
We can now shut off the old website and will the transition will be complete. However, there will still be a lot of work done.
Phase 9 - Refactor functionality
Once all the previous phases have been completed critiquecircle v2.0 is running and we have removed the legacy code. However, the functionality is much the same as on the old site, the tables and columns are named the same and there are a bunch of hacks in place to make things work the way they used to.
Now that the legacy website has been removed I will need to work on transitioning to a modern architecture. This work will probably last a long time, but we will be in a good place to do it having all the tooling and framework in place.
At this point in time we will finally see the light at the end of the tunnel and start to get in some much needed improvements.