CAESY was the brainchild of Bob Rondeau, DMD, looking to fill a need in patient education in the dental industry. From video content delivered on computer CDs to DVDs CAESY moved to a network-based delivery using a linux server and Flash front-end.
(more on the history of CAESY)

The Problem

When I arrived, the development team was demoralized and disjointed. They were working on the CAESY Enterprise 5.5 release and it was nearing completion. The software was immensely brittle, producing a new failure with nearly every change made.

The back end was written in PHP, written in the simple non-object page-response style of programming that PHP encourages (yes, it can be done better). The front end applications were hard-wired to expect that specific resources (videos and pdfs) would be available via specific database row identifiers. This led to a deep coupling danger between the applications and the database data.

The front end applications, written in C# and Flash, would retrieve server resources via PHP-served URLs. In the majority case, these calls were to a single method... "q.php" was simply a pass thru that accepted the database query to run and returned the result as text. (Indeed, it also required the database, the server and the database username and password in the clear, with every query).

Frequently, the bugs found by QA would trace back to a change made, or a change that failed to be made, in the data or in references to data that had changed. Content updates from the video production group would cause troubling openings for bugs in delivery of content files, incorporation into the database, referencing in the front end applications, and on and on...

One release, that was intended only to add content ended costing the company more than $50,000 and three months to develop! This cost could be traced back directly to the initial design failures coupling database row ids to front end application behaviors.

Problem Summary

From all these observations, it was clear that

• the base architecture of the software was deeply flawed,
• the database design was marred by initial architectural assumptions,
• the development issues were affecting the business directly, in cost of development, time-to-market, and quality of releases.

The Solution

After extensive discussions, the decision was made to re-write the system from scratch. The team agreed on a few principles for future development tools and design:

• object oriented design and programming throughout
• agile development processes
• services-oriented architecture for all communications

With these principles in hand, I tasked the team with selecting the programming language they would use for the re-write. I took this direction both to address the existing schisms within the team and because I believe strongly that developers should have a primary voice in the tools they use to implement their ideas.

A write-up of the considerations that went into the language selection would be pretty informative, I expect, but I'll put that off to another essay. Suffice to say that after considering ease-of-development, time-of-development and speed issues for linux server languages, including PHP, C#/Mono, Python and Ruby on Rails, we selected Ruby for the re-write.

The re-write took 18 months to accomplish and is now on the market as CAESY Enterprise 6.0. The server software was completely rewritten, from PHP to Ruby on Rails. The database design was simplified from a wallpaper monster down to a few easily understood pages. The clients were updated to incorporate the services oriented interaction; C# using SOAP and Flash using the WebOrb flash remoting support in Rails.

Content Management

As part of the redesign, we were anxious to reduce that failure point that had bedeviled every release of the software: the handoff from video and document production to the development team for incorporation in software releases.

We answered this need by building a complete content management system that connected to the same back end that we were shipping as the commercial server. Building the content management system ourselves, interacting directly with the shipping server allowed us to deploy directly from the server the production group managed. The video production group was thus in complete control of the versioning and quality of the content shipped. No more embarassing mistakes about the wrong version of a video making it's way into the product.

Management Challenges

CAESY brought me in as the first professional software development management they had hired. The development group had been led variously by the founder, a graphic artist, a developer without management experience and a trainer. While all had made their best effort, none could bring any development process or software team dynamics knowledge to the team. I mention all of that to prepare you for the challenges that had developed within the team.

Team Dynamics

Software teams can become quite dysfunctional. :-)

When I arrived, the software development group had been split between the "server" programmers and the "client" programmers. This was a hardened chasm, with a gulf of about ten feet between the two camps. Literally split. Enormous distrust of each side by the other. Insults and disparaging comments about the work ethic, knowledge and abilities of opposing developers.

The team's process had evolved over time, roughly centered around a waterfall methodology and large specifications documents. This followed the usual time-crunch behavior and programmer heroics that accompany the waterfall process in a small company. At the same time, it provided no real visibility into the activities and contributions of the team members.

Version control was a haphazard activity built around Microsoft's SourceSafe tool. Since the largest amount of code was Linux based, this made real version control as a pervasive discipline hard to accomplish. Because check-in activity was sporadic at best, it made activity measurement hard; most of the evaluation of programmer activity happened by verbal interviews. This added to the distrust and dysfunction in the team since no one had an objective measure of the others efforts.

Development within the Company

Development had held an odd place within CAESY. They were simultaneously the stars of the company in producing cutting edge products delivering the video content that was the real product. But they also had a reputation for shoddy work and tardy and buggy releases. Management by the lash was the only model really employed.

The development team was aware of these feelings throughout the company and demoralized by them. They felt powerless in the face of imposed deadlines and a brittle code-base. Feature sets and timelines seemed arbitrary and management capricious.

Management Solutions

These were challenging issues to overcome. Human interaction is messier and more difficult to debug than any code programmers encounter. :)

Empowerment

The first solution was something I implemented before accepting the position.

I had listened to the description of the state of the development team as delivered to me by the VP of Operations during the interview process. Based on that story, I requested a meeting with the entire development team before accepting the position. In that discussion it became clear that the constant release pressure had forced the developers to make choices that made the software progressively more brittle. The quality issues this raised would likely never be solved by trying to "detangle the spaghetti during digestion" as maintenance has sometimes been described. A full re-design and a from the ground up re-write would be necessary so I made an approval of a re-write a condition of my employment.

Aggressive Visibility

The second solution, both for the sake of the team and outward from the team to the organization, was to make the development effort completely and ruthlessly transparent.

We switched version control tools from SourceSafe to Subversion. We did this for cross-platform convenience, of course, but also because using SVN would allow us to tap the CVSWeb interface. By setting up the web visibility into the development version control activity, we made it possible for everyone, both within the team and in the company at large, to see the development effort as it occurred. We implemented a wiki to capture all development discussions, decisions and research. This provided another view into the activity of the developmen team, since not all output becomes code.

We made every development milestone visibile throughout the company. By opening the server work, the in-development client installers and the content management tools visible to the company, we allowed them to see for themselves the progress being made as well as the difficulty involved in completing feature 'a' so they could move on to feature 'b'.

Team Accountability

The team dysfunction that arose from all the years of isolated efforts and distrust were met head-on with the visibility made possible by the wiki and the code repository. We made it a team expectation that every programming effort should move from working-state to working-state and that these iterations should speed up; complete iterations should take hours not days or weeks.

Within days of this expectation settling over the group, each team member's work ethic and ability to contribute were made painfully and objectively clear. The team members who would not, or could not, contribute quickly made the decision to find another less visible place to work. To be clear, I did not make this determination, indeed, I encouraged these team members to step up, to make an effort and we would all match that effort with training and an understanding environment. Still, they chose to leave the harsh lighting of our new development process.

This turnover led to some wonderful new hires as the team reformed and put the product out.

Management Team Interaction

Interactions with the managers of each of the departments was challenging throughout the development of the 6.0 product. Immense pressure existed, mostly stemming from the pain imposed by the decision to accept that a re-write of the buggy software truly was necessary. This decision forced a change to the status-quo timeline of twice-annual releases of the software.

My response to this pressure was simple: transparency. Continous and aggressive transparency into the process of getting the software to market. At each step, we'd work through peer training about the software development environment and processes. I did a lot re-training from the modes of thought laid down by the founder's interaction with the development, software by whim, heroic (and buggy) last minute all-nighters, etc.

This wasn't always successful. Patterson, the parent company, didn't always grasp or acknowledge the realities of what we were doing. The funding for filling positions vacated by departing developers was delayed for months, impacting delivery. Late funding of those positions was followed by the incorrect belief (read the Mythical Man Month!) that time could be recaptured. I continue to consider what further steps I could have taken to improve that communication.

Closing

To close this story, for the moment... CAESY was a great experience. The development team still in place is made of stellar performers I'm thrilled to call friends. The product is outstanding. I wish CAESY well.

Skills applied in the project included:

Ruby, Ruby on Rails, Python, MySQL, Linux, Apache, UML

Please contact my references from this project!