Web Publishing Technologies

Checkout my new blog

2010-08-14T22:53:00.001-07:00

Finally, having battled with SEO, I am putting my thoughts together in this new blog. Hopefully I will have the time and energy and the prodding of few smart people to keep it going here :).

If you are in the Atlanta area, call on my brother's new Handyman biz

2010-08-14T22:35:00.001-07:00

Especially if you are commercial - a hotel, office, building or industrial enterprise needing facility management maintenance work, he is offering some really good deals to sign you up as customer at this point. The company - Crisis Solutions of Georgia has a good deal of experience in this kind of maintenance work. Especially pools, large scale equipment etc. If you own a home and have regular handyman work, you may want to give Crisis Solutions of Georgia a shot as well. Tell them I sent you there and get a discount!

Cuil - Smart or Stupid?

2010-05-05T20:34:00.000-07:00

For the last two consecutive days, we had one of our subscription sites "attacked" by the Cuil spider. The spider would come to the site during peak business hours and do requests at the rate of 1-2 requests per second on average. When we analyzed the traffic it seemed to be having some sort of conversation or dialog with the search engine on our website - feeding all kinds of queries to our website search engine. I am not sure where it was picking up the search terms from. Perhaps it was using the search terms from the search results itself to be able to reconstruct a version of the content itself. Technically you can do that and it sounds kind of cool, but it seems cheap for a VC funded west coast search engine company.

Listing a small business

2010-04-01T13:41:00.000-07:00

Starting a small business is very hard. I thought the hard part was trying to manage all the different things which go along with it. Appears to be a false assumption! I think the hardest part of running a small business (and probably any business) is to find customers who want what you can deliver or even simpler - find customers who will buy from you or use your services.

My brother started a small business - Crisis Solutions. He distributed flyers all around; then he faxed them to businesses he knew would need them. Then we redesigned the flyers and put a small website together as well. The business still works on a word of mouth basis since it is fairly new but we are working hard on some different ideas to scale it.

One of the ideas worth blogging about was getting it listed in the internet directories and yellow pages. This is something which is relatively cheap and what an entrepreneur should do and it doesn't take all that long. Consider - Yelp, Yahoo! Local, Mapquest, Google Maps, Superpages all offer some form of free listing for businesses. Then you should make sure your website is properly keyword-ed and submitted to all the different search engines.

For a business like Crisis Solutions, mail advertising also makes a lot of sense. That's what we will be looking into next...

Why you should get a Search Engine

2010-04-01T13:31:00.000-07:00

I can remember the endless debate we went through inside our organization trying to make a case for a commercial grade search engine. Google Appliance - which we were using internally was simply not doing a good job in terms of making the searches user friendly or giving us the control to do so. At that point, we were also thinking about search as a functionality for our subscribers and not as a big revenue silo in itself. We have come a long way since that. Its been 2 months...

To start with, we replaced the search engine on one of our subscription sites which was using some open source module for search (don't think Lucene - think simpler).After we did that 3-4 step searches became more like two step etc. This was because of the meta data and facets exposed by the search engine. Meta-data categories combine the best of words from plain old google like search and forms based search (where you basically specify all the criteria upfront and you get much lower results set).

Then we started replacing the search engine on our main libraries at Thompson.com. While we were doing that we realized the almost endless sales potential - like a goldmine we accidentally unearthed. We had tried numerious ways in the past to make parts of our content searchable, expose them to Google etc to get more traffic, but we had never before figured out how we can take virtually any part of our products (which are all firewalled), expose that in a search - thereby helping the user determine if they really wanted the $500 product much more easily than before. The buying decision time could get trimmed pretty heavily here. In the next month we will basically experiment with ways on how to nicely execute on this.

Then we found that using search we can much more effectively bring all our disparate web properties, databases, site content together in one place. Using additional infrastructure on the back end - like user databases and a front end reverse proxy we could effectively combine all our web properties (which exist as if they were subject to the Big Bang) in days. We are toying with that as well.

Lets see if this vision of an integrated web - Thompson Web is real or not. Using technology, everything is possible except there are a million ways to get there. Some cost millions or hundreds of thousands of dollars and others cost just a few dollars more. We need to find out what works for us...

Subdirectories v/s Subdomains in URL structure

2010-03-24T12:44:00.000-07:00

Deciding on a URL structure for your web properties is perhaps one of the most important things marketing has to do. The same problem also applies to web products which are not behind paid walls. To better know the difference between these two approaches for determining which one to pick, I'd refer you to Matt Cutts blog. Now to go on to why you should pick either one of them...

The right answer depends on how mature (or old) your website is - how much traffic you are already getting and whether the benefits of subdomains apply to your situation.

The safe choice is always using the subdirectory model. This works well for SEO and managing the website. The main reason why you may want to go to the subdomain model is that with some good keywords in the subdomain, it is slightly better for SEO. From a branding standpoint, it puts a stake in the ground as well. In other words, it sends a stronger message. Beware - there are risks involved and the good news is that you can actually do a slow migration if you decide to head in that direction.

Search engines treat subdomains as separate websites. It just doesnt finish there. They do special handling for sub-domains to detect duplicate content. Because they know that all the subdomains are usually owned by the same entity, they want to protect users from spam. So while posting content, you must be careful not to unknowingly duplicate stuff. Its always better to keep the content split in different disk areas or web servers to avoid this. Any SEO hit due to an accidental sharing of data can have a long lasting effect.

If you do want to experiment with subdomains, one method would be to start at the least risky component of the website. Note that subdomains also demand that there be substantial content for Google to see in that domain. When you split the domain off, you will loose traffic which you now need to build back two times for the ROI. On the technical side, if you own the complete server infrastructure you may want to play around with a reverse http proxy using apache or lighthttd or nginx etc which allow URLs to be mapped w/o physical changes in the network.

DITA or Docbook

2010-03-17T09:20:00.000-07:00

I attended a webinar few days back on DITA. Something I looked at last year when we decided to use DocBook instead. At the time DITA was still cutting edge with not a whole lot of support out there from an open source standpoint or companies supporting it.

It seems to me that the value of DITA is proportional to the size of the "DITA Network". An analogy of fax machines comes to mind. If you have a fax machine and I have a fax machine, the value of my fax machine just went from zero to a few bucks. It's the network effect. DITA seemed to have some attributes of "mixing" which told me that it could benefit from a network effect.

People like me would be wondering what benefit do we get at this point going from DocBook to DITA? DocBook provides us the necessary automation, standards support, community, open source and execution speed that we need. Our customers don't "see" DocBook or "DITA" in our products. All they see is how the content is laid out and how user friendly it is or what tools they have to find what they want inside the content.

Update on blocked site

2010-03-17T08:48:00.000-07:00

While we didn't hear back from Google or StopBadWare, the block placed on the site was automatically removed by the crawlers last night. That was after we removed all the references to the openads scripts. Welcome to the vagaries of Google!

Update on Google Site Block

2010-03-16T20:02:00.000-07:00

I just cannot believe that the openads javascript was in fact the reason why our site was blacklisted. Here is how it all works. Google publishes an API which is used by Firefox and Chrome to warn users when a site is blacklisted by Google. IE doesn't use that. Lucky for us - 77% of our userbase uses IE. I did find a couple of different ways in which you can find infected pages or scan those and for benefits of those who get into this quagmire, I am going to list them below:

a) Search for site:www.XXX.com in Google to get a list of pages. In this list you can eyeball the ones which are marked harmful

b) Scan your site using: http://safeweb.norton.com/

c) Scan it using http://www.unmaskparasites.com/

d) Scan it using https://wam.dasient.com/

Then to finally fix the issue, submit a request onto Google Webmaster Account and the StopBadware website. I found this link very useful in this regard as well.

Is Google trying to target OpenAds users?

2010-03-16T10:11:00.000-07:00

It seems to me that Google may have found a technical way to target websites using OpenAds server for ad management. Since yesterday our website which was using openads got flagged by Chrome and then Google as using "malicious software". The issue we identified was this piece of code:

The script is posting to a URL adserver.XXX.com which is a subdomain off the main domain of our website www.XXX.com. The interesting thing to observe is that this code pattern works fine in Firefox and IE and it was working fine in Chrome until yesterday. The offending code was one used by Open Ads which is now a competitor to Google's paid admanager service.

The Smart HR Manager Mobile Reader is coming on the iPhone!

2010-01-17T21:36:00.000-08:00

We are getting close to launching a few of our publications on the iPhone. This includes Smart HR Manager and BioWorld. Both are free applications. However Smart HR Manager is applicable to subscribers (who subscribe to our web product www.SmartHRManager.com while BioWorld contains free content even non subscribers would find useful). Interested in testing out these apps? Drop me a line ...

Hub pages v/s Site Maps

2010-01-17T21:28:00.000-08:00

If you have a lot of content on your site which Google is not including, it probably is not well connected with the rest of your site which is being crawled. Typically there are a few ways to fix it - a) connect that content to the rest of the site by links b) submit a site map to Google or c) Build hub pages. I think connecting the content may be the best thing to do but since it may involve human intervention (editing) it can be a bit slow. It should be something which should be pursued properly for fuure content and may not be a practical solution for legacy content on the site. b) Submitting a site map - though this is a recommended solution in many books, sites etc, it has never worked well for me. It seems and it would make sense that search engines may noy give enough importance to content submitted like this since typical visitors dont see it. The last option - a hub page may be worth trying. A hub page is a page which contains outbound links to other pages on the site and may have a few inbound pages coming in from the site as well. You could try putting all the site pages or the next level of missing pages in a hub page with titles/categories and include the hub page in the site's menus or headers or sidebar. I think this would provide search engines what they need to start reaching the dark ends of your site...

Getting your subscription site on Google News

2010-01-17T21:17:00.000-08:00

Past few weeks have been educational in trying to get Google News working for our subscription products.

Before the recent changes by Google News in its crawling policies, we had the news crawler crawling the free newsbriefs posted on our corporate website containing but it wouldn't crawl our subscription product sites where we would only allow teasers to be shown to visitors. Submitting the sitemap to Google News also wouldn't work. Then we had News Corp threatening to take its content out of Google News after which Google allowed subscription websites with teaser only content to be displayed as such without showing full articles to visitors coming from Google.

Now Google allows for a few options for subscription sites: (a) You can show the article the user clicksed on completely but show a subscription or login paid wall to the user for any other clicks. In this case Google News wont show a subscription label for your site in the search results. Interestingly Google has been lowering the threshold on this from requiring a a few articles to be shown for free before showing the paid wall to now only the clicked on article. (b) Google can show the "subscription" tag in front of the snippet of news from your site.

You can locate your site content in Google News by searching for site:ww.foo.com in the search box. Also the fact that Google allows more options doesn't mean that it encourages cloaking - ie., showing some other content (ie., full content) to the crawler only to show the paid wall to the Google visitor. It would be extremely dissappointing it seems to the Google reader to encounter that surprise it seems. I think that only seems logical...

Fast Flip: Another twist for the publishing industry

2009-12-03T18:12:00.001-08:00

If you read Google CEO Eric Schmidt's recent article on how Google helps the publishing industry by delivering them 1 Billion clicks, he also talks about a new google app called Fast Flip. Now fast flip appears to be the next revision of Google News in the works. Google keeps giving users what they want so why would the users ever want to go directly to the news sites to get anything.

This almost sounds like a class from MBA school where we were taught about the supplier power and buyer power in Porter's Five Forces. Google simply has too much buyer power right now over individual news companies which are too many and they compete with each other.

It seems to me that the news industry should take a good look at how they can collectively take action here. Any action by one news company simply doesnt count here. W/o any news, none of Google's apps like Fast Flip or Google News are going to work!

Keep a watch on Google's crawler

2009-12-02T21:25:00.000-08:00

Newscorp recently threatened to remove its entire news content from Google and that has led Google to revise its crawling policies two days in a row. If you didn't already know this- you could read all of WSJ articles on Google News for free because of Google's anti-cloaking requirements for the content it crawls. Anti-cloaking means that a search user must be exposed to the same content the google spider was exposed to at the time the crawl happened. So WSJ had to effectively open access to any visitor from Google News since they allowed the google crawler to index their content. Since the advertizing business model is no longer working for the news industry, this move was coming I guess.

So what's changed now?

1) Google seems to have relaxed their anti-cloaking policies a bit. At least that is what it appears to be. Now after 1 click or 5 visits the news site can show a subscribe now page to the user.

2) Google has given out specific identification for the News crawler which can come several times during the day so that people can distinguish it from the generic search crawler. News sites can get more selective. I am not sure how and what teh implications of this might turn out, but I am certianly thinking about utilizing all this soon.

Thanks to Microsoft. I dont think News Corp intends to pull content out of Google, but it is just a very real threat to which Google will have to give a good response to.

Why Search is important for Publishers

2009-11-28T23:08:00.000-08:00

If you are a publisher then its highly probable you may have heard a gazillion times how important a web search tool may be for the business but its quite possible you may not have been able to build a strong enough case for it given how much your online content revenues may be. Cheer up - you are not the only one going through the search hell - new content and products are published every day by your company on the web. You may have statistics that a typical web visitor scans 2-4 pages in each session on our website. I highly doubt that someone who doesnt already know what page they are looking for would be able to scan the product offering in that few pages. So assuming 50% of the visitors fall into this category, they are probably leaving your website w/o being able to find what they are looking for.

If you rely on a search engine like Google Mini or an open source platform like Lucene or Sphinx, you may be quite far from what is really needed to make things easy to find.

There is a secondary problem as well where people browsing through content are unable to search for their topic of interest. Same kind of problem here. However generally available search engines like Google and Lucene mostly work for this need. As a speciality publisher though, I may heavily disagree with it. A Google search engine is perhaps the worst option to pick for special enterprise content bundled and sold as subscriptions. With Lucene, if you have a budget and a development team, you can probably do anything you want with it. We figured faceted search filters was one of the most important features we wanted to have. Enterprise content is not linked up much which makes it harder to figure out document ranks. Business logic which helps determine this rankings is a must have.

Getting from InDesign to DocBook

2009-11-28T22:49:00.000-08:00

Adobe's InDesign is widely used as a layout program by publishers. For those of us who want to simultaneously update the content online on a website that makes things very tricky, especially if 95% of the content being worked on either sits in InDesign or is being worked on. using InDesign makes it extremely hard for us to work with the online version of the content.

Couple of months back we finalized our process of going from PDFs to a Docbook XML file and then transforming that to HTML. This is a very useful process since it not only helps you go from InDesign -> PDF -> HTML but also it can work with almost any kind of PDF document which may have been generated from any other layout program.

Now we are trying to finalize a process to go directly from InDesign to our Docbook XML using a Word Export. What makes this really tough are the scheduled monthly updates which must be merged back in.

We also tried writing our book completely online first using the Drupal Book module. The trouble was that the editor didn't insert all the variety of styles and when we converted the book to DocBook and then HTML it didnt have a nice structure to it. Then there is also the problem of taking that XML and converting that to a PDF using another stylesheet so it can be printed.

I have heard of CMSs and tools which work with XML directly as well so that updates can be merged in real-time, marked up and then the PDF for printing the book can be generated from that. With a lot of sunk costs in InDesign training, software etc I am not sure how you build the business case for that. After all, the customer gets the same thing and this will lead to a lot of cost savings (cost savings by themselves are the achilles heel of a business case ...). Any ideas...?

Comparing Drupal and Joomla for Digital Publishing needs

2009-11-10T20:36:00.000-08:00

At Thompson, we use a couple of platforms and technologies for our publishing needs. We have a custom built-from-scratch JSP site where we host a lot of our online content. We found that while this worked very well for some of our needs, for others we were re-inventing the wheel somewhat. So we started looking into CMSs. In the open source options, we looked at Joomla and Drupal among a lot of others and I get asked this question a lot - why Drupal and why not Joomla. Some people ask - why keep the custom built JSP platform when Drupal works for you (or vice-versa). I think this is a very interesting question and my answer to this really is that we havent felt the need to restrict ourselves to one platform. Our choice of web platform depends on the product needs and we use back end web services to tie up our infrastructure, syndicate data sources which more than makes up for having to deal with multiple platforms.

Using a CMS really cuts down the time to market for our online products from few months to weeks. We are able to follow industry standard practices in design templating (or theming), our editors are extremely happy because they feel empowered to add content anytime they want and the sites really help with stuff like SEO etc. We have developed custom Drupal modules which embed Thompson's business logic for marketing and operational analytics. We also developed custom modules to work with our digital content formats. The sort of customizations we wanted to do and the control we wanted over how these sites work - Drupal has really paid off there. I am not sure if Joomla has the architecture to allow for these customizations easily. In addition we turned on a lot of back end database syndication between our Drupal sites which allows us to create content once and expose slices of that content in different places.

The only module we have had a lot of trouble with lately is the CA (Conditional Actions) module and its interdependency with unix cron. The trouble with cron dependent tasks is that you never really know whats going on with them and its sometimes too late to find out. For example, the email reminders we set up to be sent to subscribers used to sometimes fail. While we did find the cause and fixed the issue, it was hard to discover and keep track of these reminders.

While using a CMS is great from reducing the development costs involved, it also means that you always follow certain rules, patterns and practices while implementing or changing functionality. So you do loose control somewhat compared to having written the whole site from scratch. Sometimes there can be module interactions you may not predict and the CMS could be doing more than what you need (I noticed on an online service I joined that the site was sending me role expiration notices - something the Drupal admin of the site failed to turn off). So make sure you fully understand what each module does and what it is supposed to do based on your requirements. Making the least amount of assumptions helps...

The Book Module in Drupal

2009-09-29T13:09:00.001-07:00

When I found that this module exists I was pretty excited. As an online publisher this would really fit with a lot of traditional content which we were digitizing.

The Drupal Book module may sound very sophisticated, but all it does is take the individual Drupal nodes which are part of the book and it relates them to each other (parent/child/sibling etc). Sort of puts an order into the system. This turns a random collection of pages into a book. Looks neat and sounds cool. The devil is in the details.

We have been playing around with the Book module and found that from a workflow standpoint, it has both pluses and minuses. Let me explain....

We maintain all our digital content - books is what I am talking about here in the DocBook format. These are books written originally for the paper format, maintained in tools like InDesign/InCopy or PDF/Word etc. Some may debate that DocBook is not a format etc and technically that is true, but for our discussion here, lets just call it a format to make it easy to understand. Docbook is a special XML format which preserves the structure, feature and content in the book w/o its style information. A DocBook file can then be converted to PDF, any form of HTML etc by using XSL (XML stylesheets). You can see I am laying it down here in the most layman language I can ;).

The Drupal nodes on the other hand all exist in the Drupal Database. The page text of each page in the book is stored in a text blob field part of the node. This is all HTML.

So what you would do is take the DocBook version of the book, say "Advanced Guide to Drupal" and write a script to import the different pieces into the Drupal database as nodes, simultaneously creating the relationships between these nodes so they get the book structure (Drupal uses Book Page for this, but that is just semantics). There are myriad of issues with things like footnotes, diagrams - but that is all fun stuff you will encounter and fix.

The advantage is that your book and each page of it separately is in the database. If you want you can ask the editor to directly edit the book using the Drupal Admin Interface which I am sure they will like. However if you are in the same position I am, the update will probably come in the paper format first and then you go through the process of conversion to DocBook to get a new version of the book. Note - I said "book" and not the "update". When you get the complete book, there is no marking there to identify what has changed. Then you delete the previous instance of the Drupal book in the database and load this new one. You must have path aliases and clean URLs configured else the URLs will keep changing on you (since the node numbers change each time you do the upload).

The drawback is that it is hard to point out to the user what is changed unless a separate document is loaded in the system. That's what we are doing now. The other drawback is that since you are loading/unloading the database, the script takes time to run. The search engine may also need to re-index.

Since the book is in the database, you could tag the book pages and do a lot of other fancy stuff. I am still waiting to do that though - the bang for buck!

Set up your own Drupal best practices

2009-09-25T22:17:00.000-07:00

Once you do a couple of Druapal sites, you will realize that you are doing the same things again and again thinking each time how they are done. I think Drupal has something called installation profiles and this may be something to tinker with (I haven't). However what I am talking about here probably goes a bit further - each module you install must not only be configured but also styled in some consistent way. Especially shopping carts and any other payment pages which may not vary too much between sites. Having a best practices approach with some documetation may be a good idea if you have more than few people wokring on the project. We use SVN so even checkin/checkout/export/import of modules have best practices which must be followed or you will be repeating the same mistakes and forgotten steps over and over again. Here are some of things you can document in a BCP:

1) Which modules need to be installed and how they should be configured.
2) Style guidelines. E.g shopping carts have rules for which colors typically are used in what places and how the flow needs to be, where the coupon code needs to be placed relative to the product pricing and how they need to be set up.
3) Which roles need to exist and how the permission system needs to be created to satisfy operational needs

These crazy Drupal modules

2009-09-24T21:53:00.000-07:00

Before we started using Drupal, I read in a lot of places that Drupal has a module for everything. I sort of believed it - what could we possibly want to do on the sites which many others haven't done already. After about 6-8 months of building site after site I realized we were customizing Drupal a lot and even wrote a couple of new modules - some simple and others very complex. Now almost a year into this, I can re-affirm that Drupal does seem to have a module for almost anything you have seen on more than a few sites. This really helped reduce the time to market for our products from months to days. The best part was that once you set up the system a couple of times, it becomes a relatively repeatable process with predictable results. The common administration end for all the sites (which you almost never ever have to change) is something our team got to be familiar with and now they ask for the same functionality on our older sites as well.

First test post on Drupal

2009-05-24T23:14:00.000-07:00

I have been trying to find some time to blog on Drupal and how we are using it to publish web content. Finally, it looks like I made it before Memorial day this year and as a starting post, let me tell you about why this blog is going to turn out a little different from all the other wonderful Drupal blogs out there (which I go to all the time). This blog is about how we are enabling ourselves using Drupal of course to publish and maintain a whole bunch of content sites. These sites go from http://www.schoolgrants2009.com, http://www.grantsinfocenter.com, http://compliance.thompson.com, http://www.smarthrmanager.com and many more. These sites all have a common theme: it is paid content they are after and the focus is a variety of niche markets. The nature of the content ranges from simple news blurbs to some heavy duty analysis of the problems faced by professionals in each one of their individual markets. Our focus has been to make these sites as SEO friendly as possible and make the content easy to find and read. In addition we also strive to make the sites as easy to manage and maintain as possible given the size of the markets and the multitude of sites necessary to make the financial models work.

In this blog I also invite readers to tell us about their similar experiences or suggestions we can use to improve upon what we have so far. I would also try to answer any questions about how we did certain things as best as I can...