database has to be persistent meaning that the information stored in a database has to continue to exist even after the application(s) that saved and manipulated the data have ceased to run. A database also has to provide some uniform methods that are not dependent on a specific application for accessing the information that is stored inside the database. This is a pretty liberal definition of a database. Lotus Notes calls its message stores databases and by this definition they qualify. MUMPS calls its associative storage a database and while it takes a bit of a stretch even that meets this definition. There are a number of new database technologies that include object-oriented databases and associative databases and they seem to qualify as databases under this definition too.
Text or flat binary files don't qualify as databases under this definition since only the application that created one of these files knows enough about the file's contents to make use of the information stored within the file. They meet the persistence part of the DBMS definition but not the independent access part of the definition.
Other standards like the Berkeley DB format supported by Perl Python and related languages do more or less qualify as a DBMS. While it isn't what most people think of when they think about DBMS setups it does meet both the persistence and uniform access conditions for a DBMS.
An RDBMS is a Relational Data Base Management System. This adds the additional condition that the system supports a tabular structure for the data with enforced relationships between the tables. This excludes the databases that I've listed so far since they either don't support a tabular structure at all or don't enforce relationships between tables.
Microsoft's Jet database engine qualifies as an RDBMS under this definition even though it seems like the majority of its users ignore the relational side of the engine by failing to declare foreign keys. Individual FoxPro files do not qualify because they don't have any built-in method for declaring or supporting relationships even though nearly every FoxPro system I've ever seen expects or relies on these relationships.
Most DBAs think of an RDBMS as a client/server system. The database engine runs on a server and client applications connect and request data from the server. Microsoft SQL Server Oracle DB2 (both the Z series and the UDB product) and most of the other industrial grade databases in use today use this mental model.
No discussion of RDBMS would be complete without mentioning An Introduction to Database Systems by Chris Date. This is the present incarnation of the book that originally defined the Relational Model as Edgar F. Codd defined it. You can read more at the Learning Zone.