Your company has a knowledge debt problem
Wednesday, June 16, 2010 Organization, Recruiting When your company lacks experience in tools and techniques that can make it more productive, your company has knowledge debt.
Companies tend to operate in ways that exacerbate their knowledge debt problem. Consider this fairly typical job ad:
Initech is seeking an experienced Software Engineer to join
the engineering team.
Responsibilities
* Design core, back-end software components
* Analyze and improve efficiency, scalability, and stability
of various system resources
Requirements
* M.S. Computer Science or related field preferred
* 2+ years of Java experience
* Expert in relational data modeling and query
optimization using MySQL
I would posit a guess that this company uses Java for the majority of its work and uses MySQL on the back-end. Naturally, the company wants to recruit people who share that skill set and can "jump right in" and contribute.
This mindset is fundamentally flawed. A company should be hiring for problem solving skills, programming ability, and cultural fit, not for any specific skill set.
If anything, a company should prefer candidates who are experienced in a different set of technologies. Any worthwhile programmer will be able to pick up the technologies necessary to work with the existing code base. Hiring people with different skill sets gives the team instant experience in a new set of tools for solving problems.
Recognizing knowledge debt
Every time a new language, database, tool, or technique is invented, a company accumulates knowledge debt. Unless a company actively tries to combat knowledge debt, the company will continue to use the same old tools and techniques that feel "comfortable", fall behind, and eventually look something like the companies still using Enterprise Java today.
Similar to how technical debt has interest payments in the form of manual labor and maintenance costs, knowledge debt has interest payments in the form of being less productive than possible.
You have to recognize knowledge debt as a problem before you can combat it. Every company I've been involved with has suffered from it in some form or another, especially the larger organizations. People have difficulty stepping outside their comfort zone to solve problems in different ways with different tools.
Combat knowledge debt with experimentation
You have to make a constant investment in combating knowledge debt or you're guaranteed to fall behind. One way to combat knowledge debt is to sacrifice some productivity in the short-term by experimenting with different technologies in the hope of improving productivity for the future.
Experimentation can be difficult in the face of large legacy systems. Small, relatively independent projects are the best candidates for experimentation.
For example, I first used Clojure to implement a Thrift service to collect records to a distributed filesystem. The project was independent from the rest of our code base which made it a great candidate to try out Clojure. The project taught me a great deal about Clojure, and Clojure has been a major win for my company.
The tools you use for personal productivity are some of the best candidates for experimentation. Try a new code editor like Netbeans or Emacs. If you're used to Subversion for version control, try out Git for new projects or for open source work.
Ultimately your company needs to establish a culture of experimentation and encourage people to go outside their comfort zone.
Paying off knowledge debt through experimentation can increase your technical debt. By trying out different technologies, you're diversifying your infrastructure and making it more complicated. You may need to pay off this technical debt at some point by consolidating technologies.
Combat knowledge debt with recruiting
As I've alluded to already, another great way to combat knowledge debt is to hire people with different skill sets. This way you gain the knowledge of the ins and outs of different technologies without having to make the learning investment yourself.
Of course, just because you have individuals with useful knowledge doesn't mean your company will be able to make use of that knowledge. You need mechanisms to spread awareness of those people as resources for those tools and techniques. There's no single answer on how to spread awareness, but some ideas are having regular internal presentations to present tools and techniques to the rest of the team, or for a larger company, having an internal database mapping tools and technologies to people.
Conclusion
Besides the long term productivity gains, keeping knowledge debt low can have subtler benefits. Low knowledge debt can help with recruiting since great programmers want to work with cutting edge tools and technology. Combating knowledge debt also exposes you to different ways of looking at problems that can make you more effective with older technology. As an example of the latter point, Eric Raymond has a famous quote about Lisp that "[learning Lisp] will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot."
So get out of your comfort zone and pay off your knowledge debt. You won't regret it.
You should follow me on Twitter here.
Reader Comments (48)
[...] Your company has a knowledge debt problem — thoughts from the red … [...]
[...] Your company has a knowledge debt problem — great article on why to invest in innovation and experimentation [...]
[...] Your company has a knowledge debt problem — thoughts from the red … [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
Interesting idea. I think that it's less of a "debt" and more of an "underinvestment" problem - you don't have to eventually pay it back, you just don't reap as much benefit as you could.
I think it's important to emphasize what you said about how minimizing your knowledge debt can increase your technology debt. This is absolutely true. While I think it is a great idea to always be exploring new technologies, there is something to be said for proficiency and auxiliary cost/benefit.
For instance, Rapleaf currently uses a lot of Ruby and Java. If we added a Clojure app, we may get a lot of benefit from the app itself and learning Clojure in the process, but we'll also have to be aware of the cost of integrating that into our broader systems. How do we test it in our continuous integration suite? What does it behave like in production with high load? Can it use our internal APIs? I'm sure there are reasonable answers to all these questions, but finding those answers can be an unanticipated extra pain in the process.
I think it's always great to experiment, but make sure that you choose the right time to cut off (or fully embrace) the results of your experiments so that you are actually still getting things done.
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red planet [...]
[...] Your company has a knowledge debt problem — thoughts from the red … [...]
[...] Your company has a knowledge debt problem — thoughts from the red … [...]