Currently we have a process that dumps 20 million records into regular tables in database every 15 minutes. The data is transitory and we only need to keep it for 24 hours. Since the data is transitory and it is easily recomputed, we created a database in simple recovery mode to prevent logged operations, and that's where we store this data. We also have our main database which is in full recovery mode. We created 96 tables in database, one for each 15-minute interval of the day where we store the data for the specific interval. The data is written to database by a C++ applications:1. Truncate the appropriate table (one of 96)2. Drop the indexes and PK on the table3. Write the data from the C++ app4. Recreate the PK and indexesThe data structure is fairly constant: There are roughly 2,000 widgets and about 10,000 locations. The record length for each widget/location is about 30 bytes. We generally query the data for one specific widget in all locations. Rarely we query the widgets for a specific location. The table is written to database orderly, widget-1 for all locations, widget-2 for all locations, etc. The PK is clustered on widget, location. We also have a non-clustered index on location.The requirements have changed and we now need to store one year's worth of data. I have no intention of creating 35,000+ tables to store the 15-minute data for one year, and I'm looking into using filetables for this purpose. The C++ app will write the data into filetables and we will write C# CLR functions to read the data from the filetables. We will certainly know which files to write to and read from, just like we do today for the 96 tables we write to and query from. The C# CLR knows the structure of the data and simply works with offsets and number of bytes to read from the files.The filetables are faster to read and write, and require much less space as I don't have to index the tables, as long as the storage sequence is guaranteed by the C++ application that writes the files and known to the C# CLR function that reads the files.I have several questions regarding filetables. I know that in the end testing, testing and more testing will dictate which way to go with the options below. However, I would appreciate if someone can share his/her experience with situations similar to the one I'm facing. With that, I have a few questions:1. Are inserts/deleted form filetables logged operations? That is, should the filetables be on my main database (full recovery mode) or non-logged database (simple recovery mode)? We would still be fine doing daily full dumps of the simple-mode database if that's what's needed for performance.2. What is the overhead of the filetables? For an absolute optimal querying performance, I would write one file per widget and generate 2,000 files every 15 minutes. However that creates roughly 70 million files in one year (2,000 x 96 x 366 ). They won't be all in the same folder in case we want to look at it through Windows Explorer. We will have one folder for each day of the year, and one sub-folder for each 15-minute interval, so that each sub-folder has 2,000 files, one per widget.3. The other option is to write one file-table with 20M records and the C# reads the file from using offsets and record length. That creates 35,000+ files and we can keep one folder for each day of the year, so that each folder has 96 files.Option 3 (35,000 files) is more easily manageable than Option 2 (70M files), however Option 2 is definitely faster to read data for all locations for a given widget than Option 3, since Option 3 requires to process offsets and data length. However I'm quite confident that the rare occasions where I have to read one location and all widgets, having each widget on a different file will be more time consuming.Is there anything else I should be concerned that I haven't thought about?Thanks in advance.
↧