If I may plug my own paper ( http://arxiv.org/abs/1404.5367 ) it has state-of-the-art results. To do so you need a CRF, yes, but you also need lexicons and some form of word embeddings / word clusters.

I've seen both cases, where small good dataset outperformed large noisy one, and when large noisy dataset worked better. It seems to depend on project specifics, maybe how regular are errors in the training data, how easy model is fooled to learn to do the same errors, how large is the clean dataset.

Either way, for the evaluation you should use clean data, right? So you'd have to annotate some data manually anyways, if you want evaluation to be fair. So a good start could be to both annotate data manually and generate data using a rule-based system.

When you have both, you can evaluate model trained on rule-based data both on held-out rule-based data and on manually annotated data, and check how different are results. You can also train a small model on this clean dataset and evaluate it (use cross-validation to use data efficiently), compare it to a model trained on generated data. And you can also train a model which uses a combined dataset (be careful with evaluation - e.g. use the whole rule-based dataset and 9/10 folds of a clean dataset, with larger weight, and evaluate on the rest 1/10 fold; repeat for all 10 possible splits and aggregate metrics). You can also evaluate rule-based approach directly.

There are many options once you have a clean dataset. I'd be very cautious starting project without data for validation you can trust, as you can't know if efforts you're putting in your model improve results or make them worse.