diff --git a/di/memstats/init.q b/di/memstats/init.q new file mode 100644 index 00000000..0c8d26cf --- /dev/null +++ b/di/memstats/init.q @@ -0,0 +1,3 @@ +\l ::memstats.q + +export:([objsize;memusageall;memusagevars]) diff --git a/di/memstats/memstats.md b/di/memstats/memstats.md new file mode 100644 index 00000000..751e97f9 --- /dev/null +++ b/di/memstats/memstats.md @@ -0,0 +1,57 @@ +# Memory Usage +This module can be used to calculate the approximate size of an object in memory, or for generating a table containing the approximate size of each object in memory. + +## Main funtions +The module contains two methods for calculating memory usage. + +The `memusage` functions generate a table containing the approximate memoryusage of each object in the kdb session in bytes / megabytes using -22!. This can be useful quick approximations. + +`memusagevars[]`:Generates a table of the approximate memory usage statistics of all variables in a kdb session. + +`memusageall[]`:Generates a table of the approximate memory usage statistics of all variables and views in a kdb session. + +---- + +The `objsize` function is more computationally expensive, it tries to calculate the actual memory size of an object by including nested types and attributes. + +`objsize[]`:Returns the approximate size of an individual kdb object including nested types and attributes. + +## memstats table schema +The memusage table is returned from either the `memusagevars` or `memusageall` functions. + +| Column | Type | Description | +|----------|-------------|---------------------------------------------| +| variable | `symbol` | Namespace and name of variable | +| size | `long` | The approximate size of the object in bytes | +| sizeMB | `int` | The approximatee size of the object in MB | + +## Example +Below is an example of loading the module into a session and viewing the size of different objects. + +```q +\\ Loading the module into a session +memstats: use `di.memstats + +\\ View dictionary of functions +memstats + +\\ Calculating the memory usage of an object + +a:1 / - an atom should return 16 + +b: ([]a:`a`b`c; b:1 2 3) + +memstats.objsize[a] + +memstats.objsize[b] + +// View a and b in the memstats table + +select from memstats.memusagevars[] where variable in `..a`..b + +variable size sizeMB +-------------------- +..b 69 0 +..a 17 0 + +``` \ No newline at end of file diff --git a/di/memstats/memstats.q b/di/memstats/memstats.q new file mode 100644 index 00000000..6bfa1678 --- /dev/null +++ b/di/memstats/memstats.q @@ -0,0 +1,89 @@ +/ library for viewing the approximate memory size of individual kdb objects +/ and viewing the approximate memory usage statistics of a kdb session + +/ functionality to return approximate memory size of kdb+ objects + +attrsize:{ + / `u#2 4 5 unique 32*u + $[`u=a:attr x;32*count distinct x; + / `p#2 2 1 parted (8*u;32*u;8*u+1) + `p=a;8+48*count distinct x; + 0] + }; + +/ (16 bytes + attribute overheads + raw size) to the nearest power of 2 +calcsize:{[c;s;a] `long$2 xexp ceiling 2 xlog 16+a+s*c}; + +vectorsize:{calcsize[count x;typesize x;attrsize x]}; + +/ raw size of atoms according to type, type 20h->76h have 4 bytes pointer size +typesize:{4^0N 1 16 0N 1 2 4 8 4 8 1 8 8 4 4 8 8 4 4 4 abs type x}; + +sampling:{[f;x] + / pick samples randomly accoding to threshold and apply function + threshold:100000; + $[thresholdt:type x;$[-2h=t;32;16]; + / list & enum list + t within 1 76h;vectorsize x; + / exit early for anything above 76h + 76h1000 has no attrbutes (i.e. table unlikely to have 1000 columns, list of strings unlikely to have attr for some objects only + (d[0] within 1 76h)&1=count d:distinct t;calcsize[count x;ptrsize;0]+"j"$scalesampling[{sum calcsize[count each x;typesize x 0;$[10000)&size>0),1,1,does large table show size and sizeMB + +run,0,0,q,tview::select from .test.smalltable,1,1,test memusage and views +true,0,0,q,1~count select from memstats.memusageall[] where variable in `..tview,1,1,is view shown in memusageall table +true,0,0,q,0~count select from memstats.memusagevars[] where variable in `..tview,1,1,no views shown in memusagevars +