Logical data backup and rollback using incremental capture in a distributed database

Abstract

Architecture that eliminates the need for on-disk full backups of data retaining only changes that have occurred, in a separate table. Thus, the architecture provides for incremental recovery of incremental changes in a relational database (e.g., SQL). The architecture provides improved recovery time and recovery point objectives. By using the incremental capture of changed data (e.g., in an XML format), the capability is provided to capture schema changes, query the incremental change data and efficiently restore user data to an earlier point-in-time state. Changes (e.g., insert, update and delete operations) are tracked (e.g., continuously) by a set of triggers and the incrementally captured changed rows are inserted in a data capture table (a differential change “delta” table) in a human-readable format (e.g., XML). Rollback is also provided.

Claims

1 . A computer-implemented data management system having a physical media, comprising: a capture component that captures incremental change data associated with a data operation on data in a partition of a distributed database; a tracking component that creates tracking information associated with the data operation of the incremental change data; and a storage component that stores the incremental change data and associated tracking information in a table of the partition, the incremental change data being highly available. 2 . The system of claim 1 , further comprising one or more triggers that automatically initiate capture of the incremental change data, creation of the tracking information and, storage of both the incremental change data and the tracking information in the table. 3 . The system of claim 1 , wherein the incremental change data and the tracking information are stored in the table in a human-readable format that includes a self-describing schema of the changed data in the table. 4 . The system of claim 1 , wherein the incremental change data and the tracking information are stored in the table in the same transaction as the data change is committed to the database. 5 . The system of claim 1 , wherein the incremental change data is persisted in a same partition in which the data resides. 6 . The system of claim 1 , wherein the incremental change data is highly available and searchable according to a query language. 7 . The system of claim 1 , wherein the table includes a history of changes of incremental change data, the changes associated with at least one of time a data operation occurred, time a transaction occurred, or time of row creation. 8 . The system of claim 1 , further comprising a rollback component for restoring state of the row to a previous point in time. 9 . The system of claim 1 , further comprising a retention policy component that facilitates creation and application of retention policies to the incremental change data and associated tracking information. 10 . A computer-implemented data management system having a physical media, comprising: a capture component that captures incremental change data associated with a data operation on data in a partition of a distributed database; a tracking component that creates tracking information associated with the data operation of the incremental change data; a storage component that stores the incremental change data and associated tracking information in a table of the partition; and a trigger that automatically initiates the capture, tracking, and storage of the incremental change data and the tracking information in the table. 11 . The system of claim 10 , wherein the incremental change data and the tracking information are stored in the table in a human-readable format that includes a self-describing schema of data changes in the table, the incremental change data and the tracking information are stored in the table in the same transaction as the data is committed to the database. 12 . The system of claim 10 , wherein the incremental change data is persisted in the partition in which the data resides, highly available, and searchable using a query language. 13 . The system of claim 10 , wherein the table includes a history of changes of incremental change data, the changes associated with time a data operation occurred, time a transaction occurred, and time of row creation. 14 . The system of claim 10 , further comprising a rollback component for restoring state of the data to a previous point in time, and a retention policy component that facilitates creation and application of retention policies to the incremental change data and associated tracking information. 15 . A computer-implemented data management method that employs a processor and memory, comprising: receiving a data operation related to incremental change data of data in a distributed database; capturing the incremental change data in response to the data operation; creating tracking information associated with the incremental change data; and storing the incremental change data and tracking information. 16 . The method of claim 15 , further comprising storing the incremental change data as a row in a table and in an XML format that includes a self-describing schema. 17 . The method of claim 15 , further comprising querying the incremental change data using a structured query language. 18 . The method of claim 15 , further comprising applying a retention policy to the incremental change data and tracking information. 19 . The method of claim 15 , further comprising storing a transaction timestamp and data operation timestamp in association with a transaction of the incremental change data. 20 . The method of claim 15 , further comprising applying a rollback operation to the database to roll back the data to a prior point in time based on the incremental change data.
BACKGROUND [0001] In a large distributed database system, traditional SQL (structured query language) backup and restore can be used for data protection. However, there are drawbacks. An on-disk copy of the backup of the full database is needed, which requires as much storage as the database being backed up. In addition, periodic backup of the transaction log is required for lower recovery point objective (RPO). Moreover, the recovery time objective (RTO) is poor because any restore operation, regardless of the data size involved, will need to have the full backup restored followed by applying a sequence of transaction log backup files. The process can be very time-consuming and labor-intensive, and the backup is stored in a binary format that is not directly queriable. SUMMARY [0002] The following presents a simplified summary in order to provide a basic understanding of some novel embodiments described herein. This summary is not an extensive overview, and it is not intended to identify key/critical elements or to delineate the scope thereof. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. [0003] The disclosed architecture is a cost-competitive approach that eliminates the need for on-disk full backups of data. Storage is optimized by retaining only those changes that have occurred, in a separate table. Thus, the architecture provides for incremental recovery of incremental changes in a relational database (e.g., SQL). The architecture provides improved recovery time and recovery point objectives. By using the incremental capture of changed data (e.g., in an XML format), the capability is provided to capture schema changes, query the incrementally captured data and efficiently restore user data to an earlier point-in-time state and with no downtime. [0004] Changes (e.g., insert, update and delete operations) are tracked (e.g., continuously) by a set of triggers and the incrementally captured changed rows are inserted in a data capture table (a differential change “delta” table) according to a format (e.g., XML). The format is self-describing and contains the schema for the row inside the format. [0005] Data rollback decompacts the incremental changes of the appropriate rows from the data capture table to an earlier point in time, and then overwrites the rows to production data. Insert operations are optimized to not create rows in the data capture table, but to maintain change tracking information (e.g. coordinate universal time (UTC) of insertion) in the base tables. [0006] To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings. These aspects are indicative of the various ways in which the principles disclosed herein can be practiced and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS [0007] FIG. 1 illustrates a computer-implemented data management system in accordance with the disclosed architecture. [0008] FIG. 2 illustrates an alternative embodiment of a data management system. [0009] FIG. 3 illustrates an alternative representation of a data management system. [0010] FIG. 4 illustrates a system showing modifications made to a base table for Update and Delete data operations. [0011] FIG. 5 illustrates an exemplary capture and change tracking table. [0012] FIG. 6 illustrates a computer implemented data management method in accordance with the disclosed architecture. [0013] FIG. 7 illustrates additional aspects of the method of FIG. 6 . [0014] FIG. 8 illustrates a block diagram of a computing system operable to execute logical data backup and rollback using before-image capture in accordance with the disclosed architecture. [0015] FIG. 9 illustrates a schematic block diagram of a computing environment that executes logical data backup and rollback using before-image capture. DETAILED DESCRIPTION [0016] The disclosed architecture provides the capability to efficiently capture “before-image” (incremental) data changes and change tracking information of data operations, and then to use the changes and information to efficiently rollback update/delete/insert operations. The before-image data is data to which changes have been made, but before that changed data has been processed to overwrite the production data. The storage space for the incremental data is a small fraction of the space utilized for full and transaction log backups. Moreover, the time to rollback update/delete/insert operations from a data capture table is small relative to the time to restore from full and transaction log backups. [0017] As used herein, a table is a logical relation that employs a partitioning key that controls partitioning across servers, and also employs a clustering key that controls the ordering of rows within a server. A table group is a set of tables with the same partitioning key. A row group is a set of rows in a table group having the same partitioning key value. The row group is on exactly one server, but may not be clustered. Each table group can be distributed across nodes. Each storage node is assigned ranges (partitions) of key values, and each partition is replicated for durability. [0018] The before-image data can be persisted in a XML (extensible markup language) format that contains the self-describing schema of the rows. Hence, the solution works for schema evolution. Additionally, the before-image data can be persisted in the same partition (e.g. table group), and thus, is highly available. Moreover, the before-image retention policy is managed and maintained automatically and the before-image data can be queried via traditional relational languages such as TSQL (transact structured query language (SQL)). [0019] The before-image is chosen rather than the after-image so that the changes can be applied backwards from the current data inside the partition (Undo instead of Redo), and thus, eliminates the need for a full partition backup to save storage space. [0020] Reference is now made to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding thereof. It may be evident, however, that the novel embodiments can be practiced without these specific details. In other instances, well known structures and devices are shown in block diagram form in order to facilitate a description thereof. The intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claimed subject matter. [0021] FIG. 1 illustrates a computer-implemented data management system 100 in accordance with the disclosed architecture. The system 100 includes a capture component 102 that captures incremental change data 104 associated with a data operation 106 on data 108 in a partition 110 of a distributed database. The system 100 can also include a tracking component 112 that creates tracking information 114 associated with the data operation 106 of the incremental change data 104 , and a storage component 116 that stores the incremental change data 104 and associated tracking information 114 in a table 118 of the partition 108 . The incremental change data 104 is highly available for access as desired. The capture component 102 and tracking component 112 pass data and information to the storage component 116 . [0022] The incremental change data 104 and the tracking information 114 are stored in the table 118 in a human-readable format (e.g., XML-extensible markup language) that includes a self-describing schema of rows in the table 118 . The incremental change data 104 and the tracking information 114 are stored in the table 118 in the same transaction as the data 108 is committed to the database. The incremental change data 104 is persisted in a same partition (partition 110 ) in which the data 108 resides. The incremental change data 104 is highly available and searchable according to a query language. The table 118 includes a history of changes of incremental change data 104 . The changes are associated with at least one of time a data operation occurred, time a transaction occurred, or time of row creation. [0023] FIG. 2 illustrates an alternative embodiment of a data management system 200 . The system 200 includes the capture component 102 that captures the incremental change data 104 associated with the data operation 106 on the data 108 in the partition 110 of the distributed database, the tracking component 112 that creates the tracking information 114 associated with the data operation 106 , the storage component 116 that stores the incremental change data 104 and associated tracking information 114 in the table 118 . Here, the communications between the components ( 102 , 112 , 116 , 204 , and 206 ) and the trigger(s) 202 is represented by bi-directional arrows, since there can be communications between such entities in both directions. The system 200 can also include a trigger (of one or more triggers 202 ) that automatically initiates the capture, tracking, and storage of the incremental change data 104 and the tracking information 114 in the table 118 . [0024] The incremental change data 104 and the tracking information 114 are stored in the table 118 in a human-readable format that includes a self-describing schema of rows from the base table 108 . The incremental change data 104 and the tracking information 114 are stored in the table 118 in the same transaction as the 108 data is committed to the database. The incremental change data 104 is persisted in the partition 110 in which the data 108 resides, is highly available, and is searchable using a query language. The table 118 includes a history of changes of incremental change data. The changes are associated with the time a data operation occurred (a data operation timestamp), the time a transaction occurred (a transaction timestamp), and the time of row creation (a coordinated universal time-UTC). [0025] The system 200 can further comprise a retention policy component 204 that facilitates the creation and application of retention policies to the incremental change data 104 and associated tracking information 114 and, a rollback component 206 for restoring state of the data 108 to a previous point in time. [0026] FIG. 3 illustrates an alternative representation of a data management system 300 . The system 300 (and systems 100 and 200 , for example) can be used in cloud computing environments for backup and restore functions. For each partition (of multiple replicas that include a primary replica and multiple secondary replicas) in a table group 302 in the database, a change capture and tracking table 304 (e.g., the table 118 of FIG. 1 ) is created. This table 304 (also referred to as an incremental change table or “delta” table) stores row values from the original table group 302 when certain changes are made to the original table group 302 . [0027] The system 300 can be considered as two distinct applications interfacing with the table group 302 . A backup service 306 installs the data capture and tracking table(s), the triggers 202 (e.g., SQL Update and Delete triggers), and adds three tracking columns to the base tables. These additional columns are utilized to track Insert operations. These components backup the data for the table group. A restore tool 308 allows a user to restore a previous state of table group 302 to staging database 310 . [0028] To perform the backup function, triggers 202 are installed for the Delete and Update operations on every table in each table group. The triggers 202 capture the changes made on the base row and insert a record into the data capture and tracking table 304 . These values can then be inserted back, or restored, into the original database or staging database from the data capture and tracking table 304 . When restoring, the record closest in time to the requested restore time is obtained and the change is applied back to the base row. In this way, actions made upon the database can be undone, within a set time frame, by the owners of the database. [0029] The trigger captures the change made to the base row and inserts the before-image (incremental change data) into a data capture and tracking table 304 maintained by the system. The table 304 serves as the history of changes that is used to rollback (restore) operations. The data capture and tracking table 304 is persisted in the same table group as user data, so the table 304 is highly available with multiple copies. Moreover, since the before-image is in an XML format, it is easily queriable. [0030] FIG. 4 illustrates a system 400 showing modifications made to a base table 402 for Update and Delete data operations. There are three columns which are used to track Insert, Delete, and Update operations. These columns include a UTC timestamp and two DBTS (Database Time Stamp) columns (TX DBTS and OP DBTS). The TX DBTS column tracks the larger transactional context of the Insert, Delete and Update operations, and the OP DBTS column tracks the operation of a single Insert, Delete and Update. [0031] The capture and change tracking table 304 is where data from the base table is stored before changes become permanent (overwritten on the production data). There is one active capture and change tracking table 304 per partition per table group. The columns consist of XML versions of the base table data, and the UTC and DBTS tracking columns to aid in restore operations, and other metadata related to both backup and restore. When Update and Delete operations are performed on the base table, the corresponding triggers (Delete trigger 404 and Update trigger 406 ) fire and copy the old data into the capture and change tracking table 304 . [0032] There is a view 408 on the capture and change tracking table 304 . The triggers ( 404 and 406 ) refer to this view 408 as an indirection to the most recent (active) capture and change tracking table. The capture and change tracking table 304 cannot be referenced directly because its actual name changes during cleanup of old capture and change tracking tables. [0033] An information table can be provided that stores schema versions of the base tables. This is useful for tracking schema changes that span a restore operation. The backup service 306 installs all above components, and also cleans the old (expired) capture and change tracking tables. [0034] Restore stored procedures (procs) 410 restore the partition to a previous state based on a given UTC time. The procedures 410 are installed along with the other components. One proc restores a row group and the other proc restores a whole partition. [0035] When a restore operation is performed, the three columns are utilized to track the order of previous Insert, Delete, and Update operations. These columns have default values that are filled in upon each Insert, Delete, and Update operation. The default value for each column calls a function (an intrinsic) that fills in the correct values automatically during the Insert, Delete, or Update operation. [0036] The TX DBTS allows entries in the base table and the capture and change tracking table 304 to be sorted and grouped by transactions. This uses an intrinsic that returns the DBTS of the individual operation's current transaction. On the server, this intrinsic stores the current DBTS for the entire transaction and then increments the DBTS. Any operation in a given transaction will have the same DBTS value when this intrinsic is called. Any other DBTS reference will return the incremented, current non-transactional DBTS value. [0037] The OP DBTS exposes the order of each operation, and uses an intrinsic that returns the current DBTS and increments the DBTS. This is different for every entry in the base table because each Insert operation calls the intrinsic via the default value for the column. [0038] The UTC timestamp is the current UTC time of the Insert operation that created the row. This timestamp used to aid in restore operations. Users can choose a UTC date or time to which the restore is desired. Restore determines the actual restore point based on the DBTS values using the UTC timestamp as a guide. This is utilized to restore the database to a consistent transactional state. [0039] The Insert operation is tracked in the base table using columns that are added to the base table by the system. These columns store a sequence number for the operation, the UTC time of the operation as measured at the primary replica of the partition, and a transaction identifier. (These three fields are also added to the data capture table.) No rows are inserted into the data capture table for Insert operations. This optimization reduces the storage requirements at the cost of a slight increase in complexity of the rollback logic. [0040] During rollback, the last committed state of a row (which has been saved in the data capture table) before the requested rollback time is used to overwrite the base row. [0041] There is no trigger for an Insert operation. When an insertion is made into a backed up base table, no information is stored in the capture and change tracking table. The default values for the three additional columns are filled in automatically. This new data allows the restore operation to account for, and order, Insert operations. [0042] To back up the data during an Update or Delete operation, triggers are installed by the backup service 306 for the relevant base tables. On an Update or Delete operation to the base table, the triggers store an XML representation of the pre-Update or pre-Delete base table row (without the three tracking columns). This captures the data for backup, and also the “schema” of the database. Secondly, it stores the partition key. Insert trigger-based DBTS entries do not alter the base table. Without the backup service 306 running, the triggers continue to store changed data into the capture and change tracking table 304 . [0043] The view 408 on the capture and change tracking table 304 is used so that when new capture and change tracking tables are made for a given partition within a row group, and the name of the table itself changes, the triggers do not have to change. [0044] Both the Update and Delete triggers insert the necessary backup data into an installed view. Whenever creating a new capture and change tracking table, with a new name based on its expiration date, the view is recreated to point to the new table. [0045] The backup service 306 , on initialization, reads information from configuration files, which files contain information on each database that is going to be backed up by the capture and change tracking table. The service 306 then sets up the capture and change tracking tables and triggers for the table groups. The service 306 then schedules two operations: a check for base table schema changes and upgrades, and, the cleanup of the capture and change tracking tables. [0046] To restore a partition (e.g., table group) or a row group to a particular point in time, the restore tool 308 calls one or more stored procedures 410 that install during backup service initialization or after a schema change. One procedure restores every table in the entire partition. Another procedure restores a row group within that partition. The procedures take an argument for the desired point-in-time to restore. [0047] With respect to achieving transactional consistency, during a restore (rollback) operation, there is the potential to create an inconsistent state if the desired restore point falls in the middle of a stored (backed up) transaction. For each stored change to the database, the transaction start DBTS and operation DBTS are stored for each operation in a transaction. [0048] An intrinsic TX_START_DBTS( ) is exposed that returns the DBTS of the first operation in the current transaction. If no operation had been performed by this transaction before this intrinsic is called, @@DBTS is incremented and the value recorded in the transaction record. [0049] With respect to transaction start times, observe that rolling back some of a given transaction operation, while leaving other operations of the same transaction behind, may be undesirable. Consider a system in which there are many transactions running at any given time, and each of those transactions modifies more than one row. Every value of DBTS may be in the middle of some transaction. Thus, for no value of DBTS will roll back of all operations later than this value be a correct answer. Thus, using DBTS of the operation as the sole criteria is limited. [0050] The ability is provided to roll back operations that belong to transactions that have already started rolling back. In support thereof, a transaction identifier is stored with the records. Furthermore, the transaction identifier is unique across failovers. A DBTS has this property and is guaranteed to move forward in failovers. Hence, the transaction start DBTS is a candidate for this need. [0051] Consider the following timeline for transactions T 1 , T 2 , and T 3 . [0000] Time Operation 5 5: X = 0 10 T1 starts (does something) 15 T2 starts (does something) 20 T2 updates X to 2 25 T2 commits 30 T1 updates X to 1 35 T1 commits 40 T3 starts 41 T3 updates X to 3 42 T3 commits [0052] Now, restore to DBTS=28. Since transaction T 2 has committed by then, and transaction T 1 has not, X is restored to 2. However, to accomplish this, note that the transaction T 1 update occurs after the transaction T 2 update, and yet the transaction T 1 start-time is earlier than the transaction T 2 start-time. Thus, more than the transaction start time is needed to solve this problem. [0053] Columns are employed in the main table and in the capture and change tracking table in order to optimize for an insert-heavy workload. The disclosed architecture does not involve writing a second row on inserts (or even having a trigger execute). [0054] With respect to configuring backup policy, a key customer visible policy parameter is specified in a policy file. [0055] A retention period determines how long the data is available for recovery. This basically governs the duration for which tracking data will be kept in the data capture and tracking table. Old data in the data capture table can be lazily deleted by the system. [0056] The rollback procedure is executed as a rollback command that takes as parameters the row group to be rolled back and the UTC time to roll the row group back to. (A row group is a set of rows which have the same [partition] key values.) The command is capable of overwriting production data to the state that existed at the specified rollback time. A variation of the command can roll back multiple row groups to a point in time. The command can be extended in various other ways, as well. [0057] FIG. 5 illustrates an exemplary capture and change tracking table 500 . The table 500 (similar to tables 118 and 304 ) stores a partition key and type for the base table, transaction start DBTS (TX START DBTS), operation DBTS (OP DBTS), a UTC-based timestamp (used for finding the restore points), the primary key(s) of the row in the base table (as XML), the “old” value from the base table row (an XML version of the data in the database before the Delete or Update operation), and the name of the base table from which the value and primary key columns were saved. In terms of cleanup, the name of the capture and change tracking table 500 can be the expiration date. The creation date and expiration date of the capture and change tracking table 500 can be appended to the name of the table, which eliminates the need for an extra field. [0058] Included herein is a set of flow charts representative of exemplary methodologies for performing novel aspects of the disclosed architecture. While, for purposes of simplicity of explanation, the one or more methodologies shown herein, for example, in the form of a flow chart or flow diagram, are shown and described as a series of acts, it is to be understood and appreciated that the methodologies are not limited by the order of acts, as some acts may, in accordance therewith, occur in a different order and/or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a methodology could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all acts illustrated in a methodology may be required for a novel implementation. [0059] FIG. 6 illustrates a computer implemented data management method in accordance with the disclosed architecture. At 600 , a data operation related to incremental change data of data in a distributed database is received. At 602 , the incremental change data is captured in response to the data operation. At 604 , tracking information associated with the incremental change data is created. At 606 , the incremental change data and tracking information is stored. [0060] FIG. 7 illustrates additional aspects of the method of FIG. 6 . At 700 , the incremental change data is stored as a row in a table and in an XML format that includes a self-describing schema. At 702 , the incremental change data is queried using a structured query language. At 704 , a retention policy is applied to the incremental change data and tracking information. At 706 , a transaction timestamp and data operation timestamp are stored in association with a transaction of the incremental change data. At 708 , a rollback operation is applied to the database to roll back the data to a prior point in time based on the incremental change data. [0061] As used in this application, the terms “component” and “system” are intended to refer to a computer-related entity, either hardware, a combination of software and tangible hardware, software, or software in execution. For example, a component can be, but is not limited to, tangible components such as a processor, chip memory, mass storage devices (e.g., optical drives, solid state drives, and/or magnetic storage media drives), and computers, and software components such as a process running on a processor, an object, an executable, module, a thread of execution, and/or a program. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. The word “exemplary” may be used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. [0062] Referring now to FIG. 8 , there is illustrated a block diagram of a computing system 800 operable to execute logical data backup and rollback using before-image capture in accordance with the disclosed architecture. In order to provide additional context for various aspects thereof, FIG. 8 and the following description are intended to provide a brief, general description of the suitable computing system 800 in which the various aspects can be implemented. While the description above is in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that a novel embodiment also can be implemented in combination with other program modules and/or as a combination of hardware and software. [0063] The computing system 800 for implementing various aspects includes the computer 802 having processing unit(s) 804 , a computer-readable storage such as a system memory 806 , and a system bus 808 . The processing unit(s) 804 can be any of various commercially available processors such as single-processor, multi-processor, single-core units and multi-core units. Moreover, those skilled in the art will appreciate that the novel methods can be practiced with other computer system configurations, including minicomputers, mainframe computers, as well as personal computers (e.g., desktop, laptop, etc.), hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices. [0064] The system memory 806 can include computer-readable storage such as a volatile (VOL) memory 810 (e.g., random access memory (RAM)) and non-volatile memory (NON-VOL) 812 (e.g., ROM, EPROM, EEPROM, etc.). A basic input/output system (BIOS) can be stored in the non-volatile memory 812 , and includes the basic routines that facilitate the communication of data and signals between components within the computer 802 , such as during startup. The volatile memory 810 can also include a high-speed RAM such as static RAM for caching data. [0065] The system bus 808 provides an interface for system components including, but not limited to, the system memory 806 to the processing unit(s) 804 . The system bus 808 can be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), and a peripheral bus (e.g., PCI, PCIe, AGP, LPC, etc.), using any of a variety of commercially available bus architectures. [0066] The computer 802 further includes machine readable storage subsystem(s) 814 and storage interface(s) 816 for interfacing the storage subsystem(s) 814 to the system bus 808 and other desired computer components. The storage subsystem(s) 814 can include one or more of a hard disk drive (HDD), a magnetic floppy disk drive (FDD), and/or optical disk storage drive (e.g., a CD-ROM drive DVD drive), for example. The storage interface(s) 816 can include interface technologies such as EIDE, ATA, SATA, and IEEE 1394, for example. [0067] One or more programs and data can be stored in the memory subsystem 806 , a machine readable and removable memory subsystem 818 (e.g., flash drive form factor technology), and/or the storage subsystem(s) 814 (e.g., optical, magnetic, solid state), including an operating system 820 , one or more application programs 822 , other program modules 824 , and program data 826 . [0068] The one or more application programs 822 , other program modules 824 , and program data 826 can include the entities and components of the system 100 of FIG. 1 , the entities and components of the system 200 of FIG. 2 , the entities and components of the system 300 of FIG. 3 , the entities and components of the system 400 of FIG. 4 , the table 500 , and the methods represented by the flow charts of FIGS. 6-7 , for example. [0069] Generally, programs include routines, methods, data structures, other software components, etc., that perform particular tasks or implement particular abstract data types. All or portions of the operating system 820 , applications 822 , modules 824 , and/or data 826 can also be cached in memory such as the volatile memory 810 , for example. It is to be appreciated that the disclosed architecture can be implemented with various commercially available operating systems or combinations of operating systems (e.g., as virtual machines). [0070] The storage subsystem(s) 814 and memory subsystems ( 806 and 818 ) serve as computer readable media for volatile and non-volatile storage of data, data structures, computer-executable instructions, and so forth. Computer readable media can be any available media that can be accessed by the computer 802 and includes volatile and non-volatile internal and/or external media that is removable or non-removable. For the computer 802 , the media accommodate the storage of data in any suitable digital format. It should be appreciated by those skilled in the art that other types of computer readable media can be employed such as zip drives, magnetic tape, flash memory cards, flash drives, cartridges, and the like, for storing computer executable instructions for performing the novel methods of the disclosed architecture. [0071] A user can interact with the computer 802 , programs, and data using external user input devices 828 such as a keyboard and a mouse. Other external user input devices 828 can include a microphone, an IR (infrared) remote control, a joystick, a game pad, camera recognition systems, a stylus pen, touch screen, gesture systems (e.g., eye movement, head movement, etc.), and/or the like. The user can interact with the computer 802 , programs, and data using onboard user input devices 830 such a touchpad, microphone, keyboard, etc., where the computer 802 is a portable computer, for example. These and other input devices are connected to the processing unit(s) 804 through input/output (I/O) device interface(s) 832 via the system bus 808 , but can be connected by other interfaces such as a parallel port, IEEE 1394 serial port, a game port, a USB port, an IR interface, etc. The I/O device interface(s) 832 also facilitate the use of output peripherals 834 such as printers, audio devices, camera devices, and so on, such as a sound card and/or onboard audio processing capability. [0072] One or more graphics interface(s) 836 (also commonly referred to as a graphics processing unit (GPU)) provide graphics and video signals between the computer 802 and external display(s) 838 (e.g., LCD, plasma) and/or onboard displays 840 (e.g., for portable computer). The graphics interface(s) 836 can also be manufactured as part of the computer system board. [0073] The computer 802 can operate in a networked environment (e.g., IP-based) using logical connections via a wired/wireless communications subsystem 842 to one or more networks and/or other computers. The other computers can include workstations, servers, routers, personal computers, microprocessor-based entertainment appliances, peer devices or other common network nodes, and typically include many or all of the elements described relative to the computer 802 . The logical connections can include wired/wireless connectivity to a local area network (LAN), a wide area network (WAN), hotspot, and so on. LAN and WAN networking environments are commonplace in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global communications network such as the Internet. [0074] When used in a networking environment the computer 802 connects to the network via a wired/wireless communication subsystem 842 (e.g., a network interface adapter, onboard transceiver subsystem, etc.) to communicate with wired/wireless networks, wired/wireless printers, wired/wireless input devices 844 , and so on. The computer 802 can include a modem or other means for establishing communications over the network. In a networked environment, programs and data relative to the computer 802 can be stored in the remote memory/storage device, as is associated with a distributed system. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used. [0075] The computer 802 is operable to communicate with wired/wireless devices or entities using the radio technologies such as the IEEE 802.xx family of standards, such as wireless devices operatively disposed in wireless communication (e.g., IEEE 802.11 over-the-air modulation techniques) with, for example, a printer, scanner, desktop and/or portable computer, personal digital assistant (PDA), communications satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., a kiosk, news stand, restroom), and telephone. This includes at least Wi-Fi (or Wireless Fidelity) for hotspots, WiMax, and Bluetooth™ wireless technologies. Thus, the communications can be a predefined structure as with a conventional network or simply an ad hoc communication between at least two devices. Wi-Fi networks use radio technologies called IEEE 802.11x (a, b, g, etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wire networks (which use IEEE 802.3-related media and functions). [0076] Referring now to FIG. 9 , there is illustrated a schematic block diagram of a computing environment 900 that executes logical data backup and rollback using before-image capture. The environment 900 includes one or more client(s) 902 . The client(s) 902 can be hardware and/or software (e.g., threads, processes, computing devices). The client(s) 902 can house cookie(s) and/or associated contextual information, for example. [0077] The environment 900 also includes one or more server(s) 904 . The server(s) 904 can also be hardware and/or software (e.g., threads, processes, computing devices). The servers 904 can house threads to perform transformations by employing the architecture, for example. One possible communication between a client 902 and a server 904 can be in the form of a data packet adapted to be transmitted between two or more computer processes. The data packet may include a cookie and/or associated contextual information, for example. The environment 900 includes a communication framework 906 (e.g., a global communication network such as the Internet) that can be employed to facilitate communications between the client(s) 902 and the server(s) 904 . [0078] Communications can be facilitated via a wire (including optical fiber) and/or wireless technology. The client(s) 902 are operatively connected to one or more client data store(s) 908 that can be employed to store information local to the client(s) 902 (e.g., cookie(s) and/or associated contextual information). Similarly, the server(s) 904 are operatively connected to one or more server data store(s) 910 that can be employed to store information local to the servers 904 . [0079] What has been described above includes examples of the disclosed architecture. It is, of course, not possible to describe every conceivable combination of components and/or methodologies, but one of ordinary skill in the art may recognize that many further combinations and permutations are possible. Accordingly, the novel architecture is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.

Description

Topics

Download Full PDF Version (Non-Commercial Use)

Patent Citations (99)

    Publication numberPublication dateAssigneeTitle
    US-2009064160-A1March 05, 2009Microsoft CorporationTransparent lazy maintenance of indexes and materialized views
    US-5581754-ADecember 03, 1996Xerox CorporationMethodology for managing weakly consistent replicated databases
    US-7565433-B1July 21, 2009Microsoft CorporationByzantine paxos
    US-5603024-AFebruary 11, 1997International Business Machines CorporationLossless distribution of time series data in a relational data base network
    US-2007143299-A1June 21, 2007Huras Matthew A, Vincent Timothy JCommitment of transactions in a distributed system
    US-2009144220-A1June 04, 2009Yahoo! Inc.System for storing distributed hashtables
    US-2005283644-A1December 22, 2005Microsoft CorporationEfficient changing of replica sets in distributed fault-tolerant computing system
    US-2009172142-A1July 02, 2009Hitachi, Ltd.System and method for adding a standby computer into clustered computer system
    US-2009119351-A1May 07, 2009International Business Machines CorporationMethods and Computer Program Products for Transaction Consistent Content Replication
    US-7434096-B2October 07, 2008Chicago Mercantile ExchangeMatch server for a financial exchange having fault tolerant operation
    US-5140685-AAugust 18, 1992Unisys CorporationRecord lock processing for multiprocessing data system with majority voting
    US-5796999-AAugust 18, 1998International Business Machines CorporationMethod and system for selectable consistency level maintenance in a resilent database system
    US-7483922-B1January 27, 2009International Business Machines CorporationMethods and computer program products for transaction consistent content replication
    US-5603026-AFebruary 11, 1997Xerox CorporationApplication-specific conflict resolution for weakly consistent replicated databases
    US-5819272-AOctober 06, 1998Microsoft CorporationRecord tracking in database replication
    US-7222141-B2May 22, 2007International Business Machine CorporationSystem and method for synchronizing data in multiple databases
    US-2008098045-A1April 24, 2008Oracle International CorporationTechniques for automatically tracking and archiving transactional data changes
    US-7251669-B1July 31, 2007Microsoft CorporationSystem and method for database versioning
    US-6671821-B1December 30, 2003Massachusetts Institute Of TechnologyByzantine fault tolerance
    US-7334154-B2February 19, 2008Microsoft CorporationEfficient changing of replica sets in distributed fault-tolerant computing system
    US-5440735-AAugust 08, 1995International Business Machines CorporationSimplified relational data base snapshot copying
    US-5701480-ADecember 23, 1997Digital Equipment CorporationDistributed multi-version commitment ordering protocols for guaranteeing serializability during transaction processing
    US-4714995-ADecember 22, 1987Trw Inc.Computer integration system
    US-7409460-B1August 05, 2008F5 Networks, Inc.Method and apparatus for managing network traffic
    US-7290056-B1October 30, 2007Oracle International CorporationMonitoring latency of a network to manage termination of distributed transactions
    US-5671407-ASeptember 23, 1997Xerox CorporationApplication-specific conflict detection for weakly consistent replicated databases
    US-2006129575-A1June 15, 2006Lee Myung C, Lee Mi Y, Won Jong H, Kim Myung JMethod and system for supporting XQuery trigger in XML-DBMS based on relational DBMS
    US-6401120-B1June 04, 2002Microsoft CorporationMethod and system for consistent cluster operational data in a server cluster using a quorum of replicas
    US-5799321-AAugust 25, 1998Microsoft CorporationReplicating deletion information using sets of deleted record IDs
    US-2006168011-A1July 27, 2006Microsoft CorporationFast paxos recovery
    US-2006173693-A1August 03, 2006Matan Arazi, Guy SettonComputerized trading system and methods useful therefor
    US-7249280-B2July 24, 2007Microsoft CorporationCheap paxos
    US-5577240-ANovember 19, 1996Xerox CorporationIdentification of stable writes in weakly consistent replicated databases while providing access to all writes in such a database
    US-2007100739-A1May 03, 2007Sap AgMethod and system for implementing a target group for integrated auction services on a seller's e-commerce site
    US-2009070330-A1March 12, 2009Sang Yong Hwang, Ki Hong Kim, Sung Heun Wi, Sang Kyun ChaDual access to concurrent data in a database management system
    US-2007260644-A1November 08, 2007Mats Ljungqvist, Lars-Ivar SellbergMethod for enhancing the operation of a database
    US-2004098425-A1May 20, 2004Sybase, Inc.Database System Providing Improved Methods For Data Replication
    US-2010011035-A1January 14, 2010International Business Machines CorporationMethod and System for Handling Reallocated Blocks in a File System
    US-5553279-ASeptember 03, 1996International Business Machines CorporationLossless distribution of time series data in a relational data base network
    US-6874071-B2March 29, 2005International Business Machines CorporationDatabase commit control mechanism that provides more efficient memory utilization through consideration of task priority
    US-7685171-B1March 23, 2010Emc CorporationTechniques for performing a restoration operation using device scanning
    US-5335343-AAugust 02, 1994Digital Equipment CorporationDistributed transaction processing using two-phase commit protocol with presumed-commit without log force
    US-5778350-AJuly 07, 1998Electronic Data Systems CorporationData collection, processing, and reporting system
    US-2004148289-A1July 29, 2004Oracle International CorporationOne-phase commit in a shared-nothing database system
    US-2004205414-A1October 14, 2004Roselli Drew Schaffer, Rico Blaser, Lechner Mikel CarlFault-tolerance framework for an extendable computer architecture
    US-7206805-B1April 17, 2007Oracle International CorporationAsynchronous transcription object management system
    US-6438558-B1August 20, 2002Ncr CorporationReplicating updates in original temporal order in parallel processing database systems
    US-6615256-B1September 02, 2003Microsoft CorporationQuorum resource arbiter within a storage network
    US-7107419-B1September 12, 2006Google Inc.Systems and methods for performing record append operations
    US-2005149609-A1July 07, 2005Microsoft CorporationConflict fast consensus
    US-6959323-B1October 25, 2005Lucent Technologies Inc.Scalable atomic multicast
    US-2003182328-A1September 25, 2003Jules Paquette, Martin Gagnon, Marc Boone, Dirk CandaeleApparatus and method for sharing data between multiple, remote sites of a data network
    US-7478400-B1January 13, 2009Symantec Operating CorporationEfficient distributed transaction protocol for a distributed file sharing system
    US-6970876-B2November 29, 2005Solid Information TechnologyMethod and arrangement for the management of database schemas
    US-2003115429-A1June 19, 2003International Business Machines CorporationDatabase commit control mechanism that provides more efficient memory utilization through consideration of task priority
    US-2005138081-A1June 23, 2005Alshab Melanie A., Bales Peter J., Covington Robert D., Sampson Richard A., Trotter Lisa M.Method and system for reducing information latency in a business enterprise
    US-5452445-ASeptember 19, 1995Oracle CorporationTwo-pass multi-version read consistency
    US-2005283659-A1December 22, 2005Microsoft CorporationCheap paxos
    US-2005262097-A1November 24, 2005Sim-Tang Siew Y, Fraisl Daniel JSystem for moving real-time data events across a plurality of devices in a network for simultaneous data protection, replication, and access services
    US-2005283373-A1December 22, 2005Microsoft CorporationCheap paxos
    US-7650533-B1January 19, 2010Netapp, Inc.Method and system for performing a restoration in a continuous data protection system
    US-7600221-B1October 06, 2009Sun Microsystems, Inc.Methods and apparatus of an architecture supporting execution of instructions in parallel
    US-2006136781-A1June 22, 2006Microsoft CorporationGeneralized paxos
    US-6985956-B2January 10, 2006Sun Microsystems, Inc.Switching system
    US-2005240633-A1October 27, 2005Oracle International CorporationOnline recovery of user tables using flashback table
    US-2002161889-A1October 31, 2002Rod Gamache, Massa Michael T., Sunita Shrivastava, Nishanov Gor V., Lomet David B., Bernstein Philip A., Rohit JainMethod and system for consistent cluster operational data in a server cluster using a quorum of replicas
    US-6938084-B2August 30, 2005Microsoft CorporationMethod and system for consistent cluster operational data in a server cluster using a quorum of replicas
    US-2008120298-A1May 22, 2008Microsoft CorporationParallelizing sequential frameworks using transactions
    US-2004158549-A1August 12, 2004Vladimir Matena, Magnus Eriksson, Jens JensenMethod and apparatus for online transaction processing
    US-5613113-AMarch 18, 1997International Business Machines CorporationConsistent recreation of events from activity logs
    US-6463532-B1October 08, 2002Compaq Computer CorporationSystem and method for effectuating distributed consensus among members of a processor set in a multiprocessor computing system through the use of shared storage resources
    US-2008209145-A1August 28, 2008Shyamsundar Ranganathan, Kalidas BalakrishnanTechniques for asynchronous data replication
    US-7558883-B1July 07, 2009Microsoft CorporationFast transaction commit
    US-6397352-B1May 28, 2002Oracle CorporationReliable message propagation in a distributed computer system
    US-6279032-B1August 21, 2001Microsoft CorporationMethod and system for quorum resource arbitration in a server cluster
    US-6701345-B1March 02, 2004Accenture LlpProviding a notification when a plurality of users are altering similar data in a health care solution environment
    US-7403901-B1July 22, 2008Accenture LlpError and load summary reporting in a health care solution environment
    US-2003225760-A1December 04, 2003Jarmo Ruuth, Jarmo Parkkinen, Petri Soini, Antoni WolskiMethod and system for processing replicated transactions parallel in secondary server
    US-2007130226-A1June 07, 2007Oracle International CorporationDatabase system that provides for history-enabled tables
    US-6978396-B2December 20, 2005Solid Information Technology OyMethod and system for processing replicated transactions parallel in secondary server
    US-2006036896-A1February 16, 2006Microsoft CorporationMethod and system for consistent cluster operational data in a server cluster using a quorum of replicas
    US-2002165724-A1November 07, 2002Blankesteijn Bartus C.Method and system for propagating data changes through data objects
    US-2005080801-A1April 14, 2005Vijayakumar Kothandaraman, Cuan William G., Todd ScallanSystem for transactionally deploying content across multiple machines
    US-7555516-B2June 30, 2009Microsoft CorporationFast Paxos recovery
    US-2010005124-A1January 07, 2010Robert Edward WagnerAutomated method for identifying and repairing logical data discrepancies between database replicas in a database cluster
    US-2003084038-A1May 01, 2003Verisign, Inc.Transactional memory manager
    US-2008034251-A1February 07, 2008Progress Software CorporationHigh availability via data services
    US-7603354-B2October 13, 2009Cinnober Financial Technology AbMethod for enhancing the operation of a database
    US-6845384-B2January 18, 2005Oracle International CorporationOne-phase commit in a shared-nothing database system
    US-6401136-B1June 04, 2002International Business Machines CorporationMethods, systems and computer program products for synchronization of queue-to-queue communications
    US-2003105761-A1June 05, 2003Mikael LagermanHistoric network configuration database
    US-2004083225-A1April 29, 2004Gondi Albert C., Johannes Klein, De Roo John S., Lanka Sitaram V., Sripada Ramprasad K.L.Method and apparatus for handling failures of resource managers in a clustered environment
    US-2008222159-A1September 11, 2008Oracle International CorporationDatabase system with active standby and nodes
    US-5940826-AAugust 17, 1999Unisys CorporationDual XPCS for disaster recovery in multi-host computer complexes
    US-2006090095-A1April 27, 2006Microsoft CorporationConsistent cluster operational data in a server cluster using a quorum of replicas
    US-2005198106-A1September 08, 2005Microsoft CorporationSimplified Paxos
    US-2008120299-A1May 22, 2008Microsoft CorporationParallelizing sequential frameworks using transactions
    US-2003172195-A1September 11, 2003Jonkers Henricus Bernardus MariaMethod and system for guaranteeing sequential consistency in distributed computations
    US-2008235245-A1September 25, 2008International Business Machines CorporationCommitment of transactions in a distributed system

NO-Patent Citations (0)

    Title

Cited By (48)

    Publication numberPublication dateAssigneeTitle
    CN-102393860-AMarch 28, 2012方正国际软件有限公司数据库数据迁移系统
    CN-104657364-AMay 27, 2015华为技术有限公司Log-structured database system query processing method and device
    US-2011106760-A1May 05, 2011International Business Machines CorporationSynchronizing database and non-database resources without a commit coordinator
    US-2011167037-A1July 07, 2011Siemens Product Lifecycle Management Software Inc.Traversal-free rapid data transfer
    US-2011178984-A1July 21, 2011Microsoft CorporationReplication protocol for database systems
    US-2011321012-A1December 29, 2011International Business Machines CorporationNon-Intrusive Measurement of Content Quality Using Dry Runs with Roll-back
    US-2012005154-A1January 05, 2012Johann George, Darpan Dinker, Krishnan Manavalan, O'krafka Brian WEfficient recovery of transactional data stores
    US-2012233250-A1September 13, 2012International Business Machines CorporationAuto-updatable document parts within content management systems
    US-2012246118-A1September 27, 2012International Business Machines CorporationMethod, apparatus and database system for restoring tenant data in a multi-tenant environment
    US-2012260340-A1October 11, 2012Webroot Inc.Methods and apparatus for dealing with malware
    US-2012284225-A1November 08, 2012International Business Machines CorporationAuto-updatable document parts within content management systems
    US-2014317448-A1October 23, 2014Facebook, Inc.Incremental checkpoints
    US-2015033217-A1January 29, 2015Oracle International CorporationExternal platform extensions in a multi-tenant environment
    US-2015046399-A1February 12, 2015Hitachi, Ltd.Computer system, data allocation management method, and program
    US-2015178407-A1June 25, 2015Microsoft CorporationConstructing queries for execution over multi-dimensional data structures
    US-2015324280-A1November 12, 2015International Business Machines CorporationFlash copy relationship management
    US-2015379050-A1December 31, 2015Amazon Technologies, Inc.Configurable-capacity time-series tables
    US-8332358-B2December 11, 2012Siemens Product Lifecycle Management Software Inc.Traversal-free rapid data transfer
    US-8527462-B1September 03, 2013Microsoft CorporationDatabase point-in-time restore and as-of query
    US-8751549-B2June 10, 2014International Business Machines CorporationPersistent data management in multi-image code load systems
    US-8839031-B2September 16, 2014Microsoft CorporationData consistency between virtual machines
    US-8856593-B2October 07, 2014Sandisk Enterprise Ip LlcFailure recovery using consensus replication in a distributed flash memory system
    US-8868487-B2October 21, 2014Sandisk Enterprise Ip LlcEvent processing in a flash memory-based object store
    US-8874515-B2October 28, 2014Sandisk Enterprise Ip LlcLow level object version tracking using non-volatile memory write generations
    US-8930751-B2January 06, 2015Microsoft CorporationInitializing replication in a virtual machine
    US-8954385-B2February 10, 2015Sandisk Enterprise Ip LlcEfficient recovery of transactional data stores
    US-9020905-B2April 28, 2015International Business Machines CorporationSynchronizing database and non-database resources without a commit coordinator
    US-9047351-B2June 02, 2015Sandisk Enterprise Ip LlcCluster of processing nodes with distributed global flash memory using commodity server technology
    US-9053167-B1June 09, 2015Amazon Technologies, Inc.Storage device selection for database partition replicas
    US-9075839-B2July 07, 2015International Business Machines CorporationMethod, apparatus and database system for restoring tenant data in a multi-tenant environment
    US-9092576-B2July 28, 2015International Business Machines CorporationNon-intrusive measurement of content quality using dry runs with roll-back
    US-9122740-B2September 01, 2015Siemens Product Lifecycle Management Software Inc.Bulk traversal of large data structures
    US-9135064-B2September 15, 2015Sandisk Enterprise Ip LlcFine grained adaptive throttling of background processes
    US-9164554-B2October 20, 2015Sandisk Enterprise Ip LlcNon-volatile solid-state storage system supporting high bandwidth and random access
    US-9183236-B2November 10, 2015Sandisk Enterprise Ip LlcLow level object version tracking using non-volatile memory write generations
    US-9256626-B2February 09, 2016International Business Machines CorporationMaintaining multiple copy versions of component values in a system
    US-9294482-B2March 22, 2016Oracle International CorporationExternal platform extensions in a multi-tenant environment
    US-9372889-B1June 21, 2016Amazon Technologies, Inc.Incremental statistics update
    US-9413721-B2August 09, 2016Webroot Inc.Methods and apparatus for dealing with malware
    US-9460185-B2October 04, 2016Amazon Technologies, Inc.Storage device selection for database partition replicas
    US-9471436-B2October 18, 2016Facebook, Inc.Use of incremental checkpoints to restore user data stream processes
    US-9495143-B2November 15, 2016Oracle International CorporationExternal platform extensions in a multi-tenant environment
    US-9619581-B2April 11, 2017Microsoft Technology Licensing, LlcConstructing queries for execution over multi-dimensional data structures
    US-9652495-B2May 16, 2017Siemens Product Lifecycle Management Software Inc.Traversal-free updates in large data structures
    US-9824100-B2November 21, 2017International Business Machines CorporationTechnology for importing schema changes to multiple target systems
    US-9825964-B2November 21, 2017Oracle International CorporationExternal platform extensions in a multi-tenant environment
    US-9870399-B1January 16, 2018Teradata Us, Inc.Processing column-partitioned data for row-based operations in a database system
    WO-2013038419-A1March 21, 2013Infosys LimitedIncremental download framework