Field3D
SparseFileManager Class Reference

#include <SparseFile.h>

Public Types

typedef std::list< SparseFile::CacheBlockCacheList
 

Public Member Functions

template<class Data_T >
void activateBlock (int fileId, int blockIdx)
 Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later. More...
 
float cacheEfficiency ()
 Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads. More...
 
float cacheFractionLoaded ()
 Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks) More...
 
float cacheLoadsPerBlock ()
 Computes the overall loaded-blocks-to-load ratio for cached files. More...
 
template<class Data_T >
void decBlockRef (int fileId, int blockIdx)
 Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later. More...
 
bool doLimitMemUse () const
 Returns whether to limit memory usage and do dynamic loading for sparse fields. More...
 
void flushCache ()
 Flushes the entire block cache for all files, should probably only be used for debugging. More...
 
template<class Data_T >
void incBlockRef (int fileId, int blockIdx)
 Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later. More...
 
long long numLoadedBlocks ()
 Returns the total number of blocks currently loaded into cache. More...
 
void resetCacheStatistics ()
 Resets block load. More...
 
void setLimitMemUse (bool enabled)
 Sets whether to limit memory usage and do dynamic loading for sparse fields. More...
 
void setMaxMemUse (float maxMemUse)
 Sets the maximum memory usage, in MB, by dynamically loaded sparse fields. More...
 
long long totalLoadedBlocks ()
 Returns the total number of blocks loaded (max 1 per block) into cache. More...
 
long long totalLoads ()
 Returns the total number of block loads in the cache. More...
 

Static Public Member Functions

static SparseFileManagersingleton ()
 Returns a reference to the singleton instance. More...
 

Protected Member Functions

template<class Data_T >
int getNextId (const std::string filename, const std::string layerPath)
 Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time. More...
 
template<class Data_T >
SparseFile::Reference< Data_T > * reference (int index)
 Returns a reference to the Reference object with the given index. More...
 
template<class Data_T >
void removeFieldFromCache (int refIdx)
 

Private Member Functions

void addBlockToCache (DataTypeEnum blockType, int fileId, int blockIdx)
 Adds the newly loaded block to the cache, managed by the paging algorithm. More...
 
template<class Data_T >
int64_t deallocateBlock (const SparseFile::CacheBlock &cb)
 Utility function to attempt to deallocate a single block and advance the "hand". More...
 
template<class Data_T >
void deallocateBlock (CacheList::iterator &it)
 Utility function to deallocate a single block. More...
 
void deallocateBlocks (int64_t bytesNeeded)
 Utility function to reclaim the specified number of bytes by deallocating unneeded blocks. More...
 
 SparseFileManager ()
 Private to prevent instantiation. More...
 

Private Attributes

CacheList m_blockCacheList
 List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance. More...
 
SparseFile::FileReferences m_fileData
 Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId. More...
 
bool m_limitMemUse
 Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true. More...
 
float m_maxMemUse
 Max amount om memory to use in megabytes. More...
 
int64_t m_maxMemUseInBytes
 Max amount om memory to use in bytes. More...
 
int64_t m_memUse
 Current amount of memory in use in bytes. More...
 
boost::mutex m_mutex
 Mutex to prevent multiple threads from deallocating blocks at the same time. More...
 
CacheList::iterator m_nextBlock
 Pointer to the next block to test for unloading in the cache, the "hand" of the clock. More...
 

Static Private Attributes

static SparseFileManagerms_singleton = 0
 Pointer to singleton. More...
 

Friends

template<class Data_T >
class SparseField
 

Detailed Description

Handles specifics about reading sparse fields from disk. Its primary use is to control sparse fields read using memory limiting (dynamic loading).

To enable the dynamic cache for a file, call setLimitMemUse(true) before opening the file. If you want other files to be fully loaded, call setLimitMemUse(false).

Example of how to use the cache manager to automatically unload sparse blocks from a f3d file:

SparseFileManager &sparseManager = SparseFileManager::singleton();
sparseManager.setLimitMemUse(true);  // enables cache for files to be opened 
sparseManager.setMaxMemUse(1000.0);  // sets cache to 1 GB
Field3DInputFile cacheManagedFile;
if (!cacheManagedFile.open(filename)) {
  Msg::print( "Couldn't open file: " + filename);
  return 1;
}
sparseManager.setLimitMemUse(false);  // disables cache for other files
... // You can use the file normally, loading layers and then accessing
... // with const_iterator, value(), or empty block functions like
... // getBlockEmptyValue().
... // Layers loaded from cacheManagedFile will be managed by the cache,
... // but other files you open will be fully loaded when opened because of
... // the setLimitMemUse(false) call.
Msg::print("blocks in cache: " +
  boost::lexical_cast<std::string>(sparseManager.numLoadedBlocks()));
Msg::print("cache blocks ever loaded: " +
  boost::lexical_cast<std::string>(sparseManager.totalLoadedBlocks()));
Msg::print("cache loads: " +
  boost::lexical_cast<std::string>(sparseManager.totalLoads()));
Msg::print("cache fraction loaded: " +
  boost::lexical_cast<std::string>(sparseManager.cacheFractionLoaded()));
Msg::print("cache loads per block: " +
  boost::lexical_cast<std::string>(sparseManager.cacheLoadsPerBlock()));
Msg::print("cache efficiency: " +
  boost::lexical_cast<std::string>(sparseManager.cacheEfficiency()));

If you want to flush the cache manually instead of waiting for the process to end and clean up its memory:

sparseManager.flushCache();
sparseManager.resetCacheStatistics();

Definition at line 361 of file SparseFile.h.

Member Typedef Documentation

Definition at line 371 of file SparseFile.h.

Constructor & Destructor Documentation

SparseFileManager::SparseFileManager ( )
private

Private to prevent instantiation.

Definition at line 274 of file SparseFile.cpp.

References m_blockCacheList, m_nextBlock, and setMaxMemUse().

Referenced by singleton().

275  : m_memUse(0),
276  m_limitMemUse(false)
277 {
278  setMaxMemUse(1000.0);
279  m_nextBlock = m_blockCacheList.begin();
280 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
bool m_limitMemUse
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when tr...
Definition: SparseFile.h:488
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:503
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:484
void setMaxMemUse(float maxMemUse)
Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.
Definition: SparseFile.cpp:87

Member Function Documentation

SparseFileManager & SparseFileManager::singleton ( )
static

Returns a reference to the singleton instance.

Definition at line 63 of file SparseFile.cpp.

References ms_singleton, and SparseFileManager().

Referenced by SparseField< Data_T >::addReference(), and SparseFieldIO::readData().

64 {
65  if (!ms_singleton) {
67  }
68  return *ms_singleton;
69 }
static SparseFileManager * ms_singleton
Pointer to singleton.
Definition: SparseFile.h:459
SparseFileManager()
Private to prevent instantiation.
Definition: SparseFile.cpp:274
void SparseFileManager::setLimitMemUse ( bool  enabled)

Sets whether to limit memory usage and do dynamic loading for sparse fields.

Definition at line 73 of file SparseFile.cpp.

References m_limitMemUse.

74 {
75  m_limitMemUse = enabled;
76 }
bool m_limitMemUse
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when tr...
Definition: SparseFile.h:488
bool SparseFileManager::doLimitMemUse ( ) const

Returns whether to limit memory usage and do dynamic loading for sparse fields.

Definition at line 80 of file SparseFile.cpp.

References m_limitMemUse.

Referenced by SparseFieldIO::readData().

81 {
82  return m_limitMemUse;
83 }
bool m_limitMemUse
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when tr...
Definition: SparseFile.h:488
void SparseFileManager::setMaxMemUse ( float  maxMemUse)

Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.

Definition at line 87 of file SparseFile.cpp.

References m_maxMemUse, and m_maxMemUseInBytes.

Referenced by SparseFileManager().

88 {
89  m_maxMemUse = maxMemUse;
90  m_maxMemUseInBytes = static_cast<int64_t>(m_maxMemUse * 1024 * 1024);
91 }
float m_maxMemUse
Max amount om memory to use in megabytes.
Definition: SparseFile.h:478
int64_t m_maxMemUseInBytes
Max amount om memory to use in bytes.
Definition: SparseFile.h:481
void SparseFileManager::flushCache ( )

Flushes the entire block cache for all files, should probably only be used for debugging.

Definition at line 215 of file SparseFile.cpp.

References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, m_blockCacheList, m_mutex, and m_nextBlock.

216 {
217  boost::mutex::scoped_lock lock(m_mutex);
218 
219  CacheList::iterator it = m_blockCacheList.begin();
220  while (it != m_blockCacheList.end()) {
221  SparseFile::CacheBlock &cb = *it;
222 
223  switch(cb.blockType) {
224  case DataTypeHalf:
225  deallocateBlock<half>(it);
226  break;
227  case DataTypeFloat:
228  deallocateBlock<float>(it);
229  break;
230  case DataTypeDouble:
231  deallocateBlock<double>(it);
232  break;
233  case DataTypeVecHalf:
234  deallocateBlock<V3h>(it);
235  break;
236  case DataTypeVecFloat:
237  deallocateBlock<V3f>(it);
238  break;
239  case DataTypeVecDouble:
240  deallocateBlock<V3d>(it);
241  break;
242  case DataTypeUnknown:
243  default:
244  break;
245  }
246  }
247  m_nextBlock = m_blockCacheList.begin();
248 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition: SparseFile.h:507
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:503
DataTypeEnum blockType
Definition: SparseFile.h:288
long long SparseFileManager::totalLoads ( )

Returns the total number of block loads in the cache.

Definition at line 284 of file SparseFile.cpp.

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

Referenced by cacheEfficiency(), and cacheLoadsPerBlock().

285 {
286 
287  long long int numLoads = 0;
288 
289  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
290  numLoads += m_fileData.ref<half>(i)->totalLoads();
291  }
292 
293  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
294  numLoads += m_fileData.ref<V3h>(i)->totalLoads();
295  }
296 
297  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
298  numLoads += m_fileData.ref<float>(i)->totalLoads();
299  }
300 
301  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
302  numLoads += m_fileData.ref<V3f>(i)->totalLoads();
303  }
304 
305  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
306  numLoads += m_fileData.ref<double>(i)->totalLoads();
307  }
308 
309  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
310  numLoads += m_fileData.ref<V3d>(i)->totalLoads();
311  }
312  return numLoads;
313 }
long long totalLoads()
Returns the total number of block loads in the cache.
Definition: SparseFile.cpp:284
Imath::Vec3< half > V3h
Definition: SpiMathLib.h:72
FIELD3D_NAMESPACE_OPEN typedef::half half
Definition: SpiMathLib.h:64
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Imath::V3d V3d
Definition: SpiMathLib.h:74
Imath::V3f V3f
Definition: SpiMathLib.h:73
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
size_t numRefs() const
Returns the number of file references of the corresponding collection.
long long SparseFileManager::numLoadedBlocks ( )

Returns the total number of blocks currently loaded into cache.

Definition at line 317 of file SparseFile.cpp.

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

Referenced by cacheFractionLoaded().

318 {
319 
320  long long int numBlocks = 0;
321 
322  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
323  numBlocks += m_fileData.ref<half>(i)->numLoadedBlocks();
324  }
325 
326  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
327  numBlocks += m_fileData.ref<V3h>(i)->numLoadedBlocks();
328  }
329 
330  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
331  numBlocks += m_fileData.ref<float>(i)->numLoadedBlocks();
332  }
333 
334  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
335  numBlocks += m_fileData.ref<V3f>(i)->numLoadedBlocks();
336  }
337 
338  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
339  numBlocks += m_fileData.ref<double>(i)->numLoadedBlocks();
340  }
341 
342  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
343  numBlocks += m_fileData.ref<V3d>(i)->numLoadedBlocks();
344  }
345  return numBlocks;
346 }
Imath::Vec3< half > V3h
Definition: SpiMathLib.h:72
FIELD3D_NAMESPACE_OPEN typedef::half half
Definition: SpiMathLib.h:64
long long numLoadedBlocks()
Returns the total number of blocks currently loaded into cache.
Definition: SparseFile.cpp:317
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Imath::V3d V3d
Definition: SpiMathLib.h:74
Imath::V3f V3f
Definition: SpiMathLib.h:73
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
size_t numRefs() const
Returns the number of file references of the corresponding collection.
long long SparseFileManager::totalLoadedBlocks ( )

Returns the total number of blocks loaded (max 1 per block) into cache.

Definition at line 350 of file SparseFile.cpp.

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

Referenced by cacheEfficiency(), cacheFractionLoaded(), and cacheLoadsPerBlock().

351 {
352 
353  long long int numBlocks = 0;
354 
355  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
356  numBlocks += m_fileData.ref<half>(i)->totalLoadedBlocks();
357  }
358 
359  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
360  numBlocks += m_fileData.ref<V3h>(i)->totalLoadedBlocks();
361  }
362 
363  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
364  numBlocks += m_fileData.ref<float>(i)->totalLoadedBlocks();
365  }
366 
367  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
368  numBlocks += m_fileData.ref<V3f>(i)->totalLoadedBlocks();
369  }
370 
371  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
372  numBlocks += m_fileData.ref<double>(i)->totalLoadedBlocks();
373  }
374 
375  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
376  numBlocks += m_fileData.ref<V3d>(i)->totalLoadedBlocks();
377  }
378  return numBlocks;
379 }
long long totalLoadedBlocks()
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition: SparseFile.cpp:350
Imath::Vec3< half > V3h
Definition: SpiMathLib.h:72
FIELD3D_NAMESPACE_OPEN typedef::half half
Definition: SpiMathLib.h:64
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Imath::V3d V3d
Definition: SpiMathLib.h:74
Imath::V3f V3f
Definition: SpiMathLib.h:73
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
size_t numRefs() const
Returns the number of file references of the corresponding collection.
float SparseFileManager::cacheFractionLoaded ( )

Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks)

Definition at line 383 of file SparseFile.cpp.

References numLoadedBlocks(), and totalLoadedBlocks().

384 {
385  return ((double)numLoadedBlocks())/std::max(1.0, ((double)totalLoadedBlocks()));
386 }
long long totalLoadedBlocks()
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition: SparseFile.cpp:350
long long numLoadedBlocks()
Returns the total number of blocks currently loaded into cache.
Definition: SparseFile.cpp:317
float SparseFileManager::cacheLoadsPerBlock ( )

Computes the overall loaded-blocks-to-load ratio for cached files.

Definition at line 390 of file SparseFile.cpp.

References totalLoadedBlocks(), and totalLoads().

391 {
392  return ((double)totalLoads())/std::max(1.0, ((double)totalLoadedBlocks()));
393 }
long long totalLoads()
Returns the total number of block loads in the cache.
Definition: SparseFile.cpp:284
long long totalLoadedBlocks()
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition: SparseFile.cpp:350
float SparseFileManager::cacheEfficiency ( )

Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads.

Definition at line 397 of file SparseFile.cpp.

References totalLoadedBlocks(), and totalLoads().

398 {
399  return ((double)totalLoadedBlocks())/std::max(1.0, ((double)totalLoads()));
400 }
long long totalLoads()
Returns the total number of block loads in the cache.
Definition: SparseFile.cpp:284
long long totalLoadedBlocks()
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition: SparseFile.cpp:350
void SparseFileManager::resetCacheStatistics ( )

Resets block load.

Definition at line 404 of file SparseFile.cpp.

References FIELD3D_NAMESPACE_HEADER_CLOSE, half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

405 {
406 
407  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
409  }
410 
411  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
413  }
414 
415  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
416  m_fileData.ref<float>(i)->resetCacheStatistics();
417  }
418 
419  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
421  }
422 
423  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
424  m_fileData.ref<double>(i)->resetCacheStatistics();
425  }
426 
427  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
429  }
430 }
void resetCacheStatistics()
Resets block load.
Definition: SparseFile.cpp:404
Imath::Vec3< half > V3h
Definition: SpiMathLib.h:72
FIELD3D_NAMESPACE_OPEN typedef::half half
Definition: SpiMathLib.h:64
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Imath::V3d V3d
Definition: SpiMathLib.h:74
Imath::V3f V3f
Definition: SpiMathLib.h:73
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
size_t numRefs() const
Returns the number of file references of the corresponding collection.
template<class Data_T >
void SparseFileManager::incBlockRef ( int  fileId,
int  blockIdx 
)

Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later.

Definition at line 1242 of file SparseFile.h.

References SparseFile::Reference< Data_T >::fileBlockIndices, and SparseFile::Reference< Data_T >::incBlockRef().

Referenced by SparseField< Data_T >::fastValue(), and SparseField< Data_T >::incBlockRef().

1243 {
1245 
1246  if (reference->fileBlockIndices[blockIdx] >= 0) {
1247  reference->incBlockRef(blockIdx);
1248  }
1249 }
void incBlockRef(int blockIdx)
Increment reference count on a block, indicates the block is currently in use, so prevents it from be...
Definition: SparseFile.h:731
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:119
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1197
template<class Data_T >
void SparseFileManager::decBlockRef ( int  fileId,
int  blockIdx 
)

Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later.

Definition at line 1255 of file SparseFile.h.

References SparseFile::Reference< Data_T >::decBlockRef(), FIELD3D_NAMESPACE_HEADER_CLOSE, and SparseFile::Reference< Data_T >::fileBlockIndices.

Referenced by SparseField< Data_T >::decBlockRef(), and SparseField< Data_T >::fastValue().

1256 {
1258 
1259  if (reference->fileBlockIndices[blockIdx] >= 0) {
1260  reference->decBlockRef(blockIdx);
1261  }
1262 }
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:119
void decBlockRef(int blockIdx)
Decrement reference count on a block.
Definition: SparseFile.h:740
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1197
template<class Data_T >
void SparseFileManager::activateBlock ( int  fileId,
int  blockIdx 
)

Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later.

Definition at line 1206 of file SparseFile.h.

References SparseFile::Reference< Data_T >::blockLoaded, SparseFile::Reference< Data_T >::blockMutex, SparseFile::Reference< Data_T >::blockSize(), SparseFile::Reference< Data_T >::blockUsed, SparseFile::Reference< Data_T >::fileBlockIndices, SparseFile::Reference< Data_T >::fileIsOpen(), SparseFile::Reference< Data_T >::loadBlock(), SparseFile::Reference< Data_T >::loadCounts, SparseFile::Reference< Data_T >::m_mutex, and SparseFile::Reference< Data_T >::openFile().

Referenced by SparseField< Data_T >::activateBlock(), SparseField< Data_T >::fastValue(), and SparseField< Data_T >::const_iterator::operator->().

1207 {
1209 
1210  if (reference->fileBlockIndices[blockIdx] >= 0) {
1211  if (!reference->blockLoaded[blockIdx]) {
1212  int blockSize = reference->blockSize(blockIdx);
1213  if (m_limitMemUse) {
1214  // if we already have enough free memory, deallocateBlocks()
1215  // will just return
1216  deallocateBlocks(blockSize);
1217  }
1218 
1219  if (!reference->fileIsOpen()) {
1220  reference->openFile();
1221  }
1222 
1223  boost::mutex::scoped_lock lock_A(m_mutex);
1224  boost::mutex::scoped_lock lock_B(reference->blockMutex[blockIdx]);
1225  // check to see if it was loaded between when the function
1226  // started and we got the lock on the block
1227  if (!reference->blockLoaded[blockIdx]) {
1228  reference->loadBlock(blockIdx);
1229  reference->loadCounts[blockIdx]++;
1231  m_memUse += blockSize;
1232  }
1233  }
1234  }
1235  reference->blockUsed[blockIdx] = true;
1236 }
int blockSize(int blockIdx) const
Returns the number of bytes used by the data in the block.
Definition: SparseFile.h:749
std::vector< int > blockLoaded
Whether each block is loaded. We don&#39;t use bools since vector<bool> is weird.
Definition: SparseFile.h:122
std::vector< bool > blockUsed
Flags of whether the blocks have been accessed since they were last considered for deallocation by th...
Definition: SparseFile.h:129
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:119
void openFile()
Opens the file. This is done just before the first request to loadBlock. This is delayed so that the ...
Definition: SparseFile.h:639
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition: SparseFile.h:507
bool m_limitMemUse
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when tr...
Definition: SparseFile.h:488
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:484
void addBlockToCache(DataTypeEnum blockType, int fileId, int blockIdx)
Adds the newly loaded block to the cache, managed by the paging algorithm.
Definition: SparseFile.cpp:252
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
void loadBlock(int blockIdx)
Loads the block with the given index into memory. We don&#39;t pass in a reference to where the data shou...
Definition: SparseFile.h:693
void deallocateBlocks(int64_t bytesNeeded)
Utility function to reclaim the specified number of bytes by deallocating unneeded blocks...
Definition: SparseFile.cpp:152
bool fileIsOpen()
Checks if the file used by this reference is open already.
Definition: SparseFile.h:613
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:132
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1197
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:140
template<class Data_T >
SparseFile::Reference< Data_T > * SparseFileManager::reference ( int  index)
protected

Returns a reference to the Reference object with the given index.

Definition at line 1197 of file SparseFile.h.

Referenced by SparseField< Data_T >::addReference(), SparseField< Data_T >::copySparseField(), deallocateBlock(), and SparseField< Data_T >::setupReferenceBlocks().

1198 {
1199  return m_fileData.ref<Data_T>(index);
1200 }
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
template<class Data_T >
int SparseFileManager::getNextId ( const std::string  filename,
const std::string  layerPath 
)
protected

Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time.

Definition at line 1135 of file SparseFile.h.

References SparseFile::Reference< Data_T >::create(), and SparseFile::Reference< Data_T >::m_mutex.

Referenced by SparseField< Data_T >::addReference().

1137 {
1138  using namespace SparseFile;
1139 
1140  // Must hold a mutex while appending to m_fileData
1141  boost::mutex::scoped_lock lock(m_mutex);
1142 
1143  int id = m_fileData.append<Data_T>(Reference<Data_T>::create(filename,
1144  layerPath));
1145  return id;
1146 }
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition: SparseFile.h:507
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
size_t append(typename Reference< Data_T >::Ptr ref)
Appends a reference to the collection. This is specialized so that the correct data member is accesse...
template<class Data_T >
void SparseFileManager::removeFieldFromCache ( int  refIdx)
protected

Definition at line 1152 of file SparseFile.h.

References SparseFile::Reference< Data_T >::blockLoaded, SparseFile::Reference< Data_T >::blockMutex, SparseFile::Reference< Data_T >::blocks, SparseFile::Reference< Data_T >::blockSize(), SparseFile::Reference< Data_T >::blockUsed, SparseFile::Reference< Data_T >::fileBlockIndices, SparseFile::Reference< Data_T >::loadCounts, SparseFile::Reference< Data_T >::m_mutex, SparseFile::Reference< Data_T >::refCounts, and DataTypeTraits< T >::typeEnum().

Referenced by SparseField< Data_T >::~SparseField().

1153 {
1154  boost::mutex::scoped_lock lock(m_mutex);
1155 
1158 
1159  CacheList::iterator it = m_blockCacheList.begin();
1160  CacheList::iterator end = m_blockCacheList.end();
1161  CacheList::iterator next;
1162 
1163  int64_t bytesFreed = 0;
1164 
1165  while (it != end) {
1166  if (it->blockType == blockType && it->refIdx == refIdx) {
1167  if (it == m_nextBlock) {
1168  ++m_nextBlock;
1169  }
1170  next = it;
1171  ++next;
1172  bytesFreed += reference->blockSize(it->blockIdx);
1173  m_blockCacheList.erase(it);
1174  it = next;
1175  } else {
1176  ++it;
1177  }
1178  }
1179  m_memUse -= bytesFreed;
1180 
1181  std::vector<int>().swap(reference->fileBlockIndices);
1182  reference->fileBlockIndices.resize(reference->blocks.size(), -1);
1183  typedef typename SparseFile::Reference<Data_T>::BlockPtrs BlockPtrs;
1184  BlockPtrs().swap(reference->blocks);
1185  std::vector<int>().swap(reference->blockLoaded);
1186  std::vector<bool>().swap(reference->blockUsed);
1187  std::vector<int>().swap(reference->loadCounts);
1188  std::vector<int>().swap(reference->refCounts);
1189  delete[] reference->blockMutex;
1190  reference->blockMutex = NULL;
1191 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
int blockSize(int blockIdx) const
Returns the number of bytes used by the data in the block.
Definition: SparseFile.h:749
std::vector< int > blockLoaded
Whether each block is loaded. We don&#39;t use bools since vector<bool> is weird.
Definition: SparseFile.h:122
std::vector< bool > blockUsed
Flags of whether the blocks have been accessed since they were last considered for deallocation by th...
Definition: SparseFile.h:129
std::vector< Sparse::SparseBlock< Data_T > * > BlockPtrs
Definition: SparseFile.h:109
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:119
static DataTypeEnum typeEnum()
std::vector< int > refCounts
Per-block counts of the number of current references to the blocks. If a block&#39;s ref count is non-zer...
Definition: SparseFile.h:136
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition: SparseFile.h:507
BlockPtrs blocks
Pointers to each block. This is so we can go in and manipulate them as we please. ...
Definition: SparseFile.h:125
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:503
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:484
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:132
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1197
DataTypeEnum
Definition: Traits.h:66
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:140
void SparseFileManager::addBlockToCache ( DataTypeEnum  blockType,
int  fileId,
int  blockIdx 
)
private

Adds the newly loaded block to the cache, managed by the paging algorithm.

Definition at line 252 of file SparseFile.cpp.

References m_blockCacheList, and m_nextBlock.

254 {
255  // Note: this lock is obtained while we also have a lock on the
256  // specific block (in activateBlock()), so we should make sure we
257  // never lock the SparseFileManager and *then* a block, to ensure we
258  // don't have a deadlock.
259  //
260  // Note: this was changed so the order was consistent w/ dealloc
261  // again, see activateBlock()
262  // boost::mutex::scoped_lock lock(m_mutex);
263 
264  SparseFile::CacheBlock block(blockType, fileId, blockIdx);
265  if (m_nextBlock == m_blockCacheList.end()) {
266  m_blockCacheList.push_back(block);
267  } else {
268  m_blockCacheList.insert(m_nextBlock, block);
269  }
270 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:503
void SparseFileManager::deallocateBlocks ( int64_t  bytesNeeded)
private

Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.

Definition at line 152 of file SparseFile.cpp.

References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, m_blockCacheList, m_maxMemUseInBytes, m_memUse, m_mutex, and m_nextBlock.

153 {
154  boost::mutex::scoped_lock lock_A(m_mutex);
155 
156  while (m_blockCacheList.begin() != m_blockCacheList.end() &&
157  m_maxMemUseInBytes-m_memUse < bytesNeeded) {
158 
159  if (m_nextBlock == m_blockCacheList.end())
160  m_nextBlock = m_blockCacheList.begin();
161 
163 
164  // if bytesFreed is set to >0, then we've already freed a block
165  // and advanced the "clock hand" iterator
166  int64_t bytesFreed = 0;
167 
168  switch(cb.blockType) {
169  case DataTypeHalf:
170  bytesFreed = deallocateBlock<half>(cb);
171  if (bytesFreed > 0) {
172  continue;
173  }
174  break;
175  case DataTypeFloat:
176  bytesFreed = deallocateBlock<float>(cb);
177  if (bytesFreed > 0) {
178  continue;
179  }
180  break;
181  case DataTypeDouble:
182  bytesFreed = deallocateBlock<double>(cb);
183  if (bytesFreed > 0) {
184  continue;
185  }
186  break;
187  case DataTypeVecHalf:
188  bytesFreed = deallocateBlock<V3h>(cb);
189  if (bytesFreed > 0) {
190  continue;
191  }
192  break;
193  case DataTypeVecFloat:
194  bytesFreed = deallocateBlock<V3f>(cb);
195  if (bytesFreed > 0) {
196  continue;
197  }
198  break;
199  case DataTypeVecDouble:
200  bytesFreed = deallocateBlock<V3d>(cb);
201  if (bytesFreed > 0) {
202  continue;
203  }
204  break;
205  case DataTypeUnknown:
206  default:
207  break;
208  }
209  ++m_nextBlock;
210  }
211 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
int64_t m_maxMemUseInBytes
Max amount om memory to use in bytes.
Definition: SparseFile.h:481
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition: SparseFile.h:507
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:503
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:484
DataTypeEnum blockType
Definition: SparseFile.h:288
template<class Data_T >
int64_t SparseFileManager::deallocateBlock ( const SparseFile::CacheBlock cb)
private

Utility function to attempt to deallocate a single block and advance the "hand".

Definition at line 96 of file SparseFile.cpp.

References SparseFile::CacheBlock::blockIdx, SparseFile::Reference< Data_T >::blockMutex, SparseFile::Reference< Data_T >::blockSize(), SparseFile::Reference< Data_T >::blockUsed, m_blockCacheList, m_fileData, m_memUse, m_nextBlock, SparseFile::FileReferences::ref(), SparseFile::Reference< Data_T >::refCounts, reference(), SparseFile::CacheBlock::refIdx, and SparseFile::Reference< Data_T >::unloadBlock().

97 {
98  int64_t bytesFreed = 0;
100 
101  // Note: we don't need to lock the block's mutex because
102  // deallocateBlock() is only called while the SparseFileManager's
103  // mutex is also locked (in flushCache() or deallocateBlocks()).
104  // Don't lock the block, to make sure we don't have a deadlock by
105  // holding two locks at the same time. (Because addBlockToCache()
106  // locks the manager but is also in a block-specific lock.)
107 
108  // lock the current block to make sure its blockUsed flag and ref
109  // counts don't change
110  // Note: this lock order is made consistent w/ allocate to prevent
111  // deadlocks and crashes.
112 
113  boost::mutex::scoped_lock lock_B(reference->blockMutex[cb.blockIdx]);
114 
115  // check whether the block is still in use
116  if (reference->refCounts[cb.blockIdx] > 0)
117  return bytesFreed;
118 
119  if (reference->blockUsed[cb.blockIdx]) {
120  // the block was recently used according to Second-chance paging
121  // algorithm, so skip it
122  reference->blockUsed[cb.blockIdx] = false;
123  }
124  else {
125 
126  // the block wasn't in use, so free it
127  reference->unloadBlock(cb.blockIdx);
128  bytesFreed = reference->blockSize(cb.blockIdx);
129  m_memUse -= bytesFreed;
130  CacheList::iterator toRemove = m_nextBlock;
131  ++m_nextBlock;
132  m_blockCacheList.erase(toRemove);
133  }
134  return bytesFreed;
135 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
int blockSize(int blockIdx) const
Returns the number of bytes used by the data in the block.
Definition: SparseFile.h:749
std::vector< bool > blockUsed
Flags of whether the blocks have been accessed since they were last considered for deallocation by th...
Definition: SparseFile.h:129
std::vector< int > refCounts
Per-block counts of the number of current references to the blocks. If a block&#39;s ref count is non-zer...
Definition: SparseFile.h:136
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:503
void unloadBlock(int blockIdx)
Unloads the block with the given index from memory.
Definition: SparseFile.h:712
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:484
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1197
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:140
template<class Data_T >
void SparseFileManager::deallocateBlock ( CacheList::iterator &  it)
private

Utility function to deallocate a single block.

Definition at line 140 of file SparseFile.cpp.

References SparseFile::CacheBlock::blockIdx, SparseFile::Reference< Data_T >::blockSize(), m_blockCacheList, m_fileData, m_memUse, SparseFile::FileReferences::ref(), reference(), SparseFile::CacheBlock::refIdx, and SparseFile::Reference< Data_T >::unloadBlock().

141 {
142  SparseFile::CacheBlock &cb = *it;
144  int64_t bytesFreed = reference->blockSize(cb.blockIdx);
145  m_memUse -= bytesFreed;
146  reference->unloadBlock(cb.blockIdx);
147  it = m_blockCacheList.erase(it);
148 }
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full...
Definition: SparseFile.h:499
int blockSize(int blockIdx) const
Returns the number of bytes used by the data in the block.
Definition: SparseFile.h:749
void unloadBlock(int blockIdx)
Unloads the block with the given index from memory.
Definition: SparseFile.h:712
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:492
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:484
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed...
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1197

Friends And Related Function Documentation

template<class Data_T >
friend class SparseField
friend

Definition at line 367 of file SparseFile.h.

Member Data Documentation

FIELD3D_NAMESPACE_OPEN SparseFileManager * SparseFileManager::ms_singleton = 0
staticprivate

Pointer to singleton.

Definition at line 459 of file SparseFile.h.

Referenced by singleton().

float SparseFileManager::m_maxMemUse
private

Max amount om memory to use in megabytes.

Definition at line 478 of file SparseFile.h.

Referenced by setMaxMemUse().

int64_t SparseFileManager::m_maxMemUseInBytes
private

Max amount om memory to use in bytes.

Definition at line 481 of file SparseFile.h.

Referenced by deallocateBlocks(), and setMaxMemUse().

int64_t SparseFileManager::m_memUse
private

Current amount of memory in use in bytes.

Definition at line 484 of file SparseFile.h.

Referenced by deallocateBlock(), and deallocateBlocks().

bool SparseFileManager::m_limitMemUse
private

Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true.

Definition at line 488 of file SparseFile.h.

Referenced by doLimitMemUse(), and setLimitMemUse().

SparseFile::FileReferences SparseFileManager::m_fileData
private

Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId.

Definition at line 492 of file SparseFile.h.

Referenced by deallocateBlock(), numLoadedBlocks(), resetCacheStatistics(), totalLoadedBlocks(), and totalLoads().

CacheList SparseFileManager::m_blockCacheList
private

List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance.

Definition at line 499 of file SparseFile.h.

Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), and SparseFileManager().

CacheList::iterator SparseFileManager::m_nextBlock
private

Pointer to the next block to test for unloading in the cache, the "hand" of the clock.

Definition at line 503 of file SparseFile.h.

Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), and SparseFileManager().

boost::mutex SparseFileManager::m_mutex
private

Mutex to prevent multiple threads from deallocating blocks at the same time.

Definition at line 507 of file SparseFile.h.

Referenced by deallocateBlocks(), and flushCache().


The documentation for this class was generated from the following files: