Read Only Database - Design Specification
All previous releases of the InterBase server need the database file to have "read-write" access permission (on the file system level). This is true even in the case where the user intends to use the database for "read-only" purposes. This is because, InterBase updates the transaction ID in the database header page for all current types of transactions. This inhibits the database from being used as a "read-only" database and also from prevents access to a database stored on CD-ROM where only "read" permission is only available on any file.
This design specification document provides solutions for the above stated problems. The feature implements the following:
- Enable the InterBase server to work on database files with "read" only file system permissions.
- Allow transition of database from "read-only" mode to "read-write" mode and vice-versa (not available on CD-ROM's).
- Disable certain database specific features which are only relevant for "read-write" databases, dynamically on a per-database basis for "read-only" databases. For e.g.: there is no need to allocate free-space in a data page for "read-only" databases. No room for expansion is required since update operations cannot be performed on a RO database.
When the InterBase server opens a database file, the following additional actions are performed.
- The initial Open of a database is done in RW mode. If this fails, an Open in RO mode is performed. If the Open was successful in either case, set flags to indicate that the database file has been opened in a particular mode. This will enable database file reads from CD-ROM devices and from updateable devices.
- Check the database header page whether the database is a "read-only" database and set server-side database specific (dbb) FLAG to indicate this information. If both the file structure flags AND the header page does not indicate that the DB is read-only, flag out an "invalid permission" error. This (dbb) FLAG will be used later by other InterBase sub-systems (components) during run-time, for feature enhancements. The database can be set to "read-only" using utilities such as GBAK, GFIX, InterBase Services API and JDBC API's. See Detailed Algorithm below for more information on how this is done.
Once the database is ready to be queried, INSERT/DELETE/UPDATE (and such kind of) operations are not allowed.
Exclusive access is needed in order to allow the database access mode to be toggled between RO and RW. This can be done only by the owner of the database or the SYSDBA.
Data Structures/Class Hierarchy
The existing API call isc_database_info() now accepts a new item-list option isc_info_db_read_only, which can be used to request information on whether the database is read-only or not.
A new DPB parameter isc_dpb_set_db_readonly has been introduced, to be used by utilities such as GBAK to indicate to the server to make the database RO.
New error message if parameter is not provided for i/p switch -mode (msg.gdb: Facility: GFIX, Message number: 110, PUBLIC msg, symbol: isc_gfix_mode_req)
New error message if parameter is not provided for i/p switch -mode (msg.gdb:: Facility: GBAK, Message number: 279, PUBLIC msg, symbol: isc_gbak_mode_req)
Internal Data Structures
Add dpb_db_readonly and dpb_set_db_readonly fields to the DPB structure. These are accessed in GDS_ATTACH_DATABASE() (jrd.c) and get_options() (jrd/jrd.c).
InterBase Server accessing a RO database
Setting database mode to read_only/read_write
- Set database header page to RO or RW after gaining EXCLUSIVE access to the database. Database access modes can only be changed in exclusive mode. This makes sure that there are no dirty pages in the cache that needs to be written during the change from RW to RO (not necessary in a RO to RW transition, no dirty pages in cache).
Opening a Database and set DBB flags
- Open() a database file with ReadOnly enabled, if initial Open() with ReadWrite enabled fails. Set dbb_flags in DBB structure to indicate the opened database mode.
- Educate dbb_flags about HeaderPage read_only status.
- understand the new DPB parameter isc_dpb_set_db_readonly to set a database to RO.
- Do not allow transactions to do write operations i.e INSERT/UPDATE or DELETE. Return error code isc_read_only_database in case they try to do so. This could be done both at the BLR parser and while generating DYN.
- Increment dbb_next_transaction (in-memory structure element) to identify a new transaction. We considered the alternative possibility of assigning the same transaction ID for all transactions, but rejected it for the reason that (future) services will not be able to provide information on a per-transaction (distinct) basis.
- No need to increment hdr_next_transaction in the database Header page, since the header page is not going to be written to the database.
- A new attachment ID can also be generated from the in-memory copy in DBB.
- Transactions are started read-only and pre-committed.
Cache Writer thread & Cache sub-system
- Recognize a RO database in the CCH (in-memory cache) module. Any intention to mark a page for write should be caught and an Exception raised. (to be implemented, only if needed)
- Choke the Cache Write functions while trying to Write a page in write_page(). Make the buffers clean and available for new usage.
Garbage Collector thread
- No need for Garbage collector thread to be active
GFIX - is used to set database to RO/RW, new option `-mode read_only/read_write`
- Setting a database to either RO or RW requires exclusive access to the database. For currently existing databases, users can use this option instead of backing up and restoring the database. In such a case, the server will NOT be automatically filling the data pages completely (as done while restoring using GBAK) i.e. PAG_set_no_reserve() will not be called when the database is made RO through GFIX.
- Add/Describe both the new parameters read_only and read_write to the switch -mode in alice/alice.c
- Understand new switches and set corresponding DPB parameter for the server to act on.
GBAK - new restore option `-mode read_only/read_write`
- Understand new switches for restoring database RO.
- If restoring database RO, do not leave any room in datapages for back versions.
- Backup and Restore saves/restores the RO attribute of the DB.
- If restoring database RO, while bringing database online set the database RO in the header page.
GSTAT - display "read only" status
- When GSTAT utility is used against a RO database, the "Attributes" line of "Database header page information" will display "read only".
Code changes in source modules (excluding *.h include files) to be done under #ifdef READONLY_DATABASE.
Add a new DPB parameter isc_dpb_set_db_readonly. This parameter will be used by utilities such as GBAK to indicate to the server to make the database RO.
hdr_flags from struct hdr will have a new header page flag hdr_read_only. hdr_flags is currently defined a USHORT, hence this flag will accommodate without any need to expand the structure size. This also will not need a new ODS change.
#define hdr_read_only 1024 /* Database is Read Only (RO). If not set DB is Read Write */
Convert Header page flag definitions to HEX in the include file
dbb_flags from struct dbb will have a new database block flag DBB_read_only. dbb_flags is currently defined a ULONG, hence this flag will accommodate without any need to expand the structure size.
#define DBB_read_only 0x4000L /* Database is Read Only (RO). If not set DB is Read Write */ #define DBB_being_opened_read_only 0x8000L /* Database is being opened RO. If not set, DB is being opened RW */
New field in struct dbb, SLONG dbb_attachment_id; This will give the in-memory representation of the next attachment id for that database, since the header page (hdr_attachment_id) will not be referenced for this.
prototype for PAG_set_db_readonly()
Setting database mode to read_only/read_write
Add dpb_db_readonly and dpb_set_db_readonly fields to the DPB structure. These are accessed in GDS_ATTACH_DATABASE() and get_options()
get_options() understands the new DPB parameter isc_dpb_set_db_readonly
call PAG_set_db_readonly() from jrd8_attach_database() (jrd/jrd.c), after gaining EXCLUSIVE access to the database. Use CCH_exclusive() to attach in exclusive mode.
create new function PAG_set_db_readonly(dbb, flag) (jrd/pag.c). This will imprint the new access mode (either RO or RW) into the database header page. Note: When changing the mode from RO to RW, this should reset the in-memory flag to indicate that the database is no more a RO database, thus allowing the CCH subsystem to perform a write operation on the database header page.
Check whether hdr_read_only is set in the header page and set corresponding in-memory data structures, namely DBB->dbb_flags in PAG_header() to indicate that a Database is RO i.e DBB_read_only and reset the DBB_being_opened_read_only bit. Also, handle the case where the database file permission says it has ReadOnly access and the Header Page says "No way!". Punt out with gds__no_priv error.
Opening a Database and set DBB flags
In function PIO_open()
if (desc=open(filename, RW)==-1) then desc=open (filename, RO); /* RW failed, attempt RO mode */ if (ReadOnly) then dbb->dbb_flags |=DBB_being_opened_read_only
In function PIO_open()
if (desc=CreateFile (filename, RW)==-1) then desc=CreateFile (filename, RO); /* RW failed, attempt RO mode */ if (ReadOnly) then dbb->dbb_flags |=DBB_being_opened_read_only
return without doing anything in INI_update_database(). A RO DB does not require any minor ODS upgrade.
Do not start Garbage collector thread in VIO_init() for a RO DB.
Increment in-memory transaction ID (dbb_next_transaction) in bump_transaction_id() and return value. No need to update Header page!
TRA_set_state(): Do not set the transaction state on disk inventory page, but DO set it in in-memory page (TPC_set_state()).
TRA_cleanup(): Return without doing anything for a RO DB.
TRA_header_write(): should not write to the header page. Is any action required to SYNC Header and DBB TIDs?
Some other functions TRA_xxxx() in this module also needs to be changed.
PAG_attachment_id() will get new attachment ID from dbb->dbb_attachment_id instead of hdr_attachment_id, since the header page is not updateable.
PAG_set_xxxx() functions have been modified to not allow header page changes (through GFIX) for a ReadOnly database. This is to prevent users from setting values (buffers, SYNC/ASYNC etc.) in the header page
attachment->att_flags |=ATT_no_cleanup; in GDS_ATTACH_DATABASE() in order to inhibit garbage collection.
CCH_init() needs no change, including memory_init()
cache_writer() can return immediately, thereby terminating Cache Writer thread for a RO database.
CCH_mark() needs to raise an exception (BUGCHECK (302)=="unexpected page change") if it is called for a RO database. This will enable us to catch illegal (intention to write) operations on a RO database. The macro CCH_MARK and CCH_MARK_MUST_WRITE finally call into CCH_mark() and thus will also be caught.
write_page() needs to be modified to NOT call PIO_write() and CCH_write_all_shadows() in case of a RO DB. This may not be required if we raise an exception in CCH_mark(), as above. We could raise the same exception (here too, just) as a catch-all case. Corresponding bcb_flags need to reset to indicate that the buffers are now clean.
Recognize a RO database in RLCK_reserve_relation(). This function is called whenever a relation has to be reserved (for say, writing). Since write operations (INSERT/DELETE/UPDATE) are not allowed in a RO database, this might be the perfect place to catch such executions. This also releaves us from making all the transactions RO. SQLCODE (-817) is returned back to the client. The error is "attempted update on read-only database". All operations that are not caught in the BLR parser or DYN generator (as mentioned below) should be caught here.
Catch DML statements (INSERT/DELETE/UPDATE) in parse() for BLR types blr_store, blr_modify, blr_erase etc. and raise an exception (SQLCODE -817) as above. Also, do the same for blr_gen_id & blr_set_generator for non-zero values. Currently, this has not been implemented. The check in RLCK_reserve_relation() takes care of all DML statements, albeit during execution instead of during the parse phase. Will be looked at as an enhancement later on.
Disallow increment operations for generators in DPM_gen_id()
In EXT_file(), open external files in RO mode for RO databases.
In EXT_store(), disallow "INSERT" operations for a RO database.
In BLB_create2() and delete_blob(), disallow blob creations and deletions respectively. But, let BLOB selections be allowed for RO databases.
Raise exception (SQLCODE -817, error isc_read_only_database) in DDL_generate(). This will catch all Metadata change operations.
Introduce the new definition for isc_info_db_read_only
Handle isc_info_db_read_only in INF_database_info()
Define sw_mode. Add IN_SW_ALICE_MODE, ALICE_SW_MODE_RO and ALICE_SW_MODE_RW definitions.
Add new error message for i/p switch "-mode read_only/read_write" as for other switches in GFIX. (msg.gdb: Facility: GFIX, Message number: 109)
Add new field ua_read_only as BOOLEAN to USER_ACTION structure to denote whether "read_only" option was specified in the gfix command-line.
- Set ua_read_only based on i/p switch validation.
- Add new error message if parameter is not provided for i/p switch -mode (msg.gdb: Facility: GFIX, Message number: 110, PUBLIC msg, symbol: isc_gfix_mode_req)
build_dpb() (alice/exe.c) needs to set the corresponding DPB parameter isc_dpb_set_db_readonly based on ua_read_only setting above.
Add IN_SW_BURP_MODE definition.
Call into isc_spb_res_mode of Services API for restore operation switch -mode. Services API needs to define the respective service parameter block attributes.
att_db_read_only attribute declaration, required to save a RO attribute from the header page during a backup. Used during restore to enable database as RO or RW based on attribute.
Understand new switch IN_SW_BURP_MODE in main() and BURP_gbak()
- Check for the "read_only" option in i/p and set isc_dpb_no_reserve DPB parameter in the create_database() call; The InterBase server calls PAG_set_no_reserve() (jrd/pag.c) and does not leave any free space in a data page while filling it up. In the default mode, InterBase leaves some space for back version expansion of records in the same data page. This is equivalent to the gfix -use command.
- Once the restore operation is done in RESTORE_restore(), at the end of that function set the RO flag in the database header page. This operation can be done while setting the database online.
Convert all READ_WRITE open() calls in to READ open() calls. There is actually no requirement for these database files to be opened in READ_WRITE mode on any database. This is irrespective of whether the DB is read/write or not. The function of GSTAT is only to provide statistical o/p and not to change anything in the database.
Modify PPG_print_header() to display database read/write "Attributes" as mentioned above.
Performance improvements (Not yet implemented)
Do not load triggers into in-memory Cache for a RO database.
return without loading triggers in MET_load_triggers()
No need to traverse back versions of records in a RO database.
No need to chase record versions in VIO_chase_record_version(). Return a qualified and valid record as soon as you find one. No need to invoke garbage collection on records which have back versions either.
TCS test series V4_TRANS_0 has some test cases which test different transaction modes. A similar set of tests would have to be run against a RO database. Also, new tests have to be written to exercise the new options to utilities GBAK and GFIX.
Possible Future work (needs to be researched, beyond InterBase 6.0 / project scope)
- Could implement in-memory operation of "what-if?" condition queries. This could involve UPDATE/DELETE operations, but will be volatile operations. These operations will not be reflected in the database file.
- Could a Shadow database be set to RO (mirror-site/Replication needs)?
- Different scenarios of RO database usage...
- Transcient InterBase install and run (CD-ROM Learning solutions? etc.)
- Run InterBase from CD. (create temp. files on system's temporary storage)
- Run from Shareable Network Device
- Co-existance with already installed/running InterBase (Disk .vs. CD-ROM)
- User licensing issues.
- "find" FTS file creation in Win-NT/95 automatically?