flat text files vs databases

[madMoney]

I do a lot of web coding, and often have to switch between using flat files and databases depending on the server, language, etc...

When I use PHP or CF, I use databases because of the amazing support built into both languages (and the organization of databases in general), but when I use Perl, I almost always use text files (DBI is too in depth for quick scripts imo, and I don't use Perl enough to get really familiar with it).

My question is, what are the advantages, speed and HD space -wise, of databases over text files? I understand that databases are much more extensible and organized, especially for really big collections of data, and are usually faster due to indexing, but is this always true even for very small amounts of data? Is the ease of text files (simplicity) ever worth it? when is it not?

[iDxMan]

Before posting, I did a quick google search ( http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=perl+text+vs+database ) and there was quite an array of responses, but the overall theme seemed to be pro-db.

structure, index, search, security, ease of data manipulation, etc.. (yeah, its pretty simple to open a text file, add/mod/delete data then rewrite, but still - SQL is easier)

As an example, I've written a budget app (? ~medium size app) in perl and php. Two years ago it was quite a scramble in all areas, which resulted in a rushed perl hack with flat files. Maybe it was just the rush [or lack of experience I now have?], although this year I basically had the same amount of time but it was with php and mysql. Not having to think about opening/reading/storing/locking/writing flat files was really great. I could spend more time focusing on what the app needed to do.(yeah, and already knowing 120% of what it needed to do was helpful too) It was also very handy to create a new db user and give some user-admins select/update access (via phpMyAdmin) to certain tables. No need to write more custom code as I may have had to do with flat files.

Skipping back, another theme with the above search results is speed. As the file grows in size, your performance rapidly decreases. Whereas with a properly configured database, you can have a massive amount of data without the performance hit. HD space usage? Generally text files aren't exactly efficient.


That's not to say that you should always use a DB. I don't, but I'll at least go through the following questions to myself:

* app size?
* How much data and of what kind? (dead simple or complex relationships)
* Does this need to be portable across systems that may not have access to a db? .. or that have lazy bofh's that don't install the mysql libs? :)

For small things it may not matter, but its no fun shooting yourself in the foot.

-r