remote cpu monitoring using node-kstat

The node-kstat add-on module for node.js allows you to poll specific system kstats from within node, and get back a nice array of objects with keys corresponding to the various kstat fields (like what you get from the kstat(1) command). Bryan’s repository also includes an example of the mpstat(1) command written in JavaScript that utilizes the node-kstat add-on. Given the ease with which you can implement web services using node (and supporting libraries), I thought I’d try exposing the kstat facility as a RESTful web service, and re-tool the mpstat example to run in the browser as a client.

Part 1: Exposing a kstat web service

Thanks to the express framework for node, this turned out to be the easiest part of the exercise. I wanted to expose roughly the same API as the node-kstat Reader() constructor method (which takes kstat class, module, name, and instance – all optional), but I also wanted to allow multiple kstats to be specified (within a single module). The reason for this is that mpstat(1) consumes 2 kstats (cpu::sys and cpu::vm), and I didn’t want to have to make multiple HTTP requests for each round of output from the mpstat client. I settled on the following API for my web service:
http://server.example.com:port/kstats/module/instance/name

The ‘module’ field is required and must match a single kstat module. The ‘instance‘ field may be an integer of a single instance, or the ‘*’ wildcard to indicate all instances. The ‘name‘ field may be one or more kstat names separated by semicolons. So, the URI needed to gather the 2 kstats consumed by mpstat ends up being:
http://server.example.com:port/kstats/cpu/*/vm;sys

The web service responds with JSON. The response contains an object for each of the named kstats (vm and sys), each of which contains an array with one element per CPU instance.

Here’s all the code for the kstat web service:


var express = require('express');
var kstat = require('kstat');

var app = module.exports = express.createServer();

// Routes

app.get('/kstats/:module/:instance/:name', function(req, res){

        var filter = {};

        filter["module"] = req.params.module;

        if (req.params.instance ==! '*')
                filter["instance"] = parseInt(req.params.instance, 10);

        //handle multiple semicolon-separated values for stat 'name'
        var stats = req.params.name.split(';');

        var results = {};

        var reader = {};

        for (var i=0; i < stats.length; i++) {
                var stat = stats[i];

                filter["name"] = stat;

                reader[stat] = new kstat.Reader(filter);

                results[stat] = reader[stat].read();
        }

        // Set response header to enable cross-site requests
        res.header('Access-Control-Allow-Origin', '*');

        res.send(results);

});

// Only listen on $ node app.js

if (!module.parent) {
  app.listen(3000);
  console.log("kstat server listening on port %d", app.address().port);
}

Part 2: porting mpstat.js to the browser

I really didn’t have to change much to the original server-side mpstat.js to get it working in the browser as a client to the kstat web service. The big change was to make it event-driven so I could use the Ajax model for making HTTP requests and responding to HTTP response events. To do this, I added an update_kstats() function for the HTTP response handler to call. Rather than having the mpstat object push output, I added a method to retrieve the current output from the object’s internal state.

The modified mpstat.js code (renamed mpstat_init.js) can be viewed here.

The Ajax page that includes the above mpstat_init.js can be viewed here.

(Edit: Live Demo no longer available)

Part 3: adding some flare

If you’re like most people, you like using your visual cortex to process data. Since we are running JavaScript in the browser anyway, why not take a page from the cloud analytics book, and use flot to graph some of the data we’re getting from mpstat? In the example below, I’m graphing individual CPU utilization (usr + sys columns) per-cpu, and I’m color-coding the mpstat output with the graph. The only change needed to the mpstat module was to add a method for retrieving the current output state in object form rather than lines of text (so it could be fed to the graphing library).

Source for the Ajax page (which uses the flot library) can be found here.

(Edit: Live Demo no longer available)
(Edit: 3/23/13 Source for the demo is available in a self-contained express app hosted here: https://github.com/mharsch/mpstat-browser.)

5 Tips for the first-time iPhone App developer

Having just completed the process of developing my first iPhone App, I thought I’d share some tips that I picked up along the way while they’re still fresh in my mind (i.e. still licking my wounds).

1.) Learn Objective-C and the iOS SDK in a structured way
Stanford has done a great job packaging their iOS programming course for mass consumption via iTunesU. The material is presented in a logical progression that does not assume any previous experience with Objective-C or any Apple development environment (though you do need a strong handle on object-oriented design concepts). Lectures are interleaved with demos that introduce new concepts by-example. I’ve found this lecture/demo format very effective for quickly ramping-up on the platform. I did buy a book to supplement the iTunesU course, but I never used it. Apple’s online documentation will become your new best friend, and between the class and developer.apple.com, no additional documentation is needed (ok, maybe stackoverflow.com).

2.) Design your model, prototype, then iterate on details
Everything is keyed off your model, so design it first before diving in to View Controllers. You can iterate the model design of course, but big decisions should be made up-front (like whether or not to use Core Data). I’d suggest saving Core Data for your second project, as it adds a fair amount of complexity.

I found that it was easy to bog down on trying to get each little piece to display or function as you want it to. In order to keep your momentum, I’d suggest prototyping all your features first, without worrying about look-and-feel. The SDK provides tons of functionality with very little code when you opt for their defaults. As soon as you start saying things like: “I wish that button were a different color” or “could it slide in from the other side?” — you will be forced to start descending levels of the stack and write custom code that overrides the SDK convenience features. This may be necessary to achieve your objective, but this kind of customization should be saved for the last stage (iteration).

3.) Keep a constant eye on performance
I took a lot for granted while developing my first App. One mistake that came back to bite me was relying too heavily on the iOS Simulator (part of the development environment) to gauge the app’s performance. I reached the end of my development effort and was surprised to discover that the app had horrible performance on real-world hardware (specifically iPhone 3). My first app used a large custom scroll view as the central feature (allowing the user to horizontally scroll through a list of pages, like the ‘Weather’ app that comes with the phone). Scrolling through the pages is an animation effect that the phone must generate when you put your finger down and start dragging it. The amount of work that the graphics handling layers must do is a function of several factors including how many subviews must be rendered to produce the next frame in the animation. My initial design had 6-8 subviews (a combination of labels, images, buttons, background, etc.). This proved to be way too many for the phone to handle, and thus the framerate during animation was very poor – producing a choppy effect. I was able to work out the biggest factor (too many subviews) and fix it by merging the subviews that didn’t change into one layer (the background image). Apple’s debugging tools helped with this diagnosis (specifically the ‘Core Animation’ Instrument), but I should have caught the problem much earlier in the development cycle. I now keep my old iPhone 3 on hand to use as a ‘lowest common denominator’ spot check whenever I change anything that could have a performance impact.

4.) Learn and master Apple’s provisioning scheme early on
The hoops that one must jump through to get code running on a device are many and varied. Joining the Apple Developer program ($99/yr.) is necessary to be able to push your app to a device. Once you’re signed up for the developer program, you have access to Apple’s Provisioning Portal where you generate certificates, associate phone IDs with certain profiles, etc. It’s all very complicated and not very well documented. It’s worth your time to dig into this process and really learn how certificates, profiles, App IDs, and Device IDs combine and interact to allow your app to be deployed to other phones (either through Ad-Hoc distribution, or via the iTunes App Store). A solid understanding of this mechanism will be very nice to have when it comes time to submit your App to the iTunes App Store for review (you don’t want to be scrambling at that point).

I had a small team (3 other team members) that were providing feedback on builds as the app was taking shape. Since I was the only developer, I couldn’t rely on others having the XCode development environment. To facilitate distribution of builds, it was necessary to create an ‘Ad-Hoc Provisioning Profile’ and associate that profile with builds. Once the profile was setup and working, I could generate .ipa files from XCode and my teammates could just drag them into iTunes and update their phones from there. This delivery system worked very well, and as I mentioned – it forced me to learn about the provisioning scheme early on which paid dividends later.

5.) Consider adding Analytics
It’s astoundingly easy to add analytics functionality by dropping in the Flurry SDK package. Doing so will allow you to track user sessions (launches of your app) and events that occur during app usage (clicking buttons, searching, whatever). There are obvious tradeoffs to consider here: The Flurry package is not open source, so you can’t be sure exactly what they’re doing with your data. Also, linking to their library introduces a risk factor since Apple can change their policies regarding privacy which could force you to update your app with a new Flurry SDK version (this has happened to Flurry users at least once already). With all of that said, the appeal of having analytics (for basically no investment on your part) is undeniable.

If you do decide to include Flurry Analytics in your app, I’d suggest that you utilize Blocks and GCD to dispatch calls to FlurryAPI on low priority threads (separate from the main UI thread). Since users don’t see any output from these network connections, there’s no reason to make them suffer their latency.

visualizing filesystems with parallel sets

Every so often, I run out of disk space. This inevitably leads me to take on a search-and-destroy mission: eliminate large files to free up space. Having spent some time recently studying data visualization techniques, I thought it would be an interesting exercise to try out different graphical tools to see which can best answer the question at hand: “Where are the files that are hogging my disk space?”.

I was very amused to find out that the (now common) data visualization known as a treemap was invented specifically to solve this very problem. The treemap uses area to represent the numerical value of interest (in this case: disk space utilization). The filesystem hierarchy can also be encoded by the grouping and coloring of the subdivided rectangles, though this isn’t demonstrated well by the trivial example here. Consider a 3 level directory structure that contains files totaling 100MB. The structure is as follows:

If the 100MB was evenly distributed across all the 3rd level directories, the treemap would look like this:

However, if the distribution was not uniform, the treemap will show us (using area) which directories use a disproportionate amount of space:

In the above graphic, we see that a3 is consuming 50% of the total area, while b2 and b3 consume 30% and 12% respectively. If this was a picture of my hard drive, I’d know to concentrate my file deletion efforts in directory a3 first, then b2 and so on.

Parallel Sets
An interesting data visualization technique called Parallel Sets offers (perhaps) a more flexible alternative to the treemap for representing the breakdown of a data set by categories. Let’s use the same directory example as above. A Parallel Sets view of the evenly distributed case might look something like this:

This view shows higher level directories as an aggregation of their sub-directories – a nice effect that is less obvious in the treemap approach. The effect even more pronounced in the asymmetric example (using the same values as the treemap above):

Looking at this view, I can clearly see that a3 and b2 are the low hanging fruit in my search for space hogs. We can eliminate the root level directory since it doesn’t add much value:

Adding more dimensions
At this point It’s probably a toss-up as to which approach is better. Both pictures answer the original question about which directories are contributing most to the space consumption problem. The treemap however is reaching it’s limits in terms of expressing more data, whereas the parallel sets are just getting warmed up. Let’s switch to a slightly more realistic directory tree example:

Let us also change our original question and instead consider file count instead of file size as the unit of measure. The resulting pictures would look similar, except wider ribbons mean more files rather than more space consumed.

Now let’s add the dimension “file type” where type {executable, data, text}.

Now we can see the data both converge and fan-out according to the extra property that we’ve specified. From the diagram, we can learn the following:

  • The contents of /usr/bin and /usr/lib are all executable, while /usr/include is all text
  • /var/tmp has text and data files, while /var/adm is all text
  • User ‘mharsch’ has a bit of everything in his home directory, while user ‘root’ has just text.

This same layering can be done with any attribute that is categorical (i.e. fits into a discrete number of buckets). Imagine breaking down all files by when they were last accessed (hour, day, week, month, year). Or bucketizing file size to see which directories contain many small files vs few large files.

Since the data is independent of the filesystem hierarchy, we don’t have to constrain ourselves to the tree view if it doesn’t make sense for the arbitrary question we want to ask. For example, if we want to know the relative breakdown of files owned by user ‘jsmith’ vs ‘jdoe’ broken down by file type and access history, we might end up looking at a picture like this:

By altering the order of specified attributes, you can greatly affect the complexity of the resulting image. By choosing the variable “file type” followed by “last accessed” and finally “user”, we get a much busier (and less helpful) visual:

Going beyond the filesystem
Since computer systems are full of related, categorical data – why constrain ourselves to the filesystem? Let’s look at a measurement of cpu utilization over a certain interval. During the measurement, 2 processes were running: cpuhog and failread. Each process spent some of it’s time in user mode and some in kernel mode. If we wanted to get a graphical sense of how the cpu spent it’s time during the interval, we could use combinations of stacked time-series graphs (the approach used by Analytics):

Or we could apply our new tool Parallel Sets to the problem and get something like this:

Both visuals tell the story that cpuhog spends most of it’s time running in user mode, while failread is spending most if it’s time in kernel mode. The Analytics version puts the data in the context of a timeline, while the parallel sets image does not. For the purposes of spotting trends over time, the Analytics version would be the way to go. If a historical context wasn’t important (say, for a point-in-time snapshot) the parallel sets view may be a stronger choice, since it combines the data into one graph and uses slightly less ink.

Conclusion
I think Parallel Sets has a future in visualizing certain aspects of computer systems. The tools available now are primitive (only accepting CSV data, and not allowing empty fields), but I think the tool is powerful enough that developers will be compelled to integrate it into their GUIs.

Update (1/1/2011): I just realized that the filesystem hierarchy diagrams (which were png files created using GraphVis) were not showing up on IE or Firefox. I’ve fixed these files, and they should work on all browsers now. If you read this post before using a browser other than Safari, you missed out on 2 diagrams. My apologies.

portable build system for apps with DTrace

Having recently been bitten by this issue, I thought I’d share a trivial example showing how to use Autoconf to stage a platform-independent build system for DTrace-enabled applications.

Before going any further, I should call out some pre-requisite reading: Adam covers usage of USDT probes here, while Dave digs deeper into the subject here. Autoconf is not intuitive, so if you haven’t worked with it before (as I hadn’t until a few days ago), you should read chapters 1 and 2 of John Calcote’s excellent online reference Autotools: a practitioners guide….

For those making use of USDT probes in applications, there are some differences in the steps needed to bake-in DTrace support at compile time (depending on the underlying platform). In this post I’ll discuss how to use GNU Autoconf to deliver a portable build system that will ‘do the right thing’ on supported platforms, or allow you to disable DTrace on unsupported platforms.

Differences between Solaris and Mac OS X
Solaris requires that a provider object file be generated (using dtrace(1M) with the -G flag) and linked to the final executable. As explained in Adam’s post, the steps are as follows:


$ dtrace -h -s my_provider.d
$ gcc -c my_app.c 
$ dtrace -G -s my_provider.d my_app.o 
$ gcc -o my_app my_provider.o my_app.o

OS X doesn’t require this intermediate step (in fact, dtrace(1M) on OS X doesn’t have a ‘-G’ flag). So, on a Mac, the steps for accomplishing the same thing look like this:


$ dtrace -h -s my_provider.d
$ gcc -o my_app my_app.c

Of course, we can split out the compiling and linking steps (perhaps more realistic for anything beyond a toy example):


$ dtrace -h -s my_provider.d
$ gcc -c -o my_app.c
$ gcc -o my_app my_app.o

Looking at what needs to be run on each platform, we can see that our Makefiles will have to be different. A Makefile for Solaris might look like this:


my_app: my_app.o my_provider.o
        gcc -o my_app my_app.o my_provider.o
my_app.o: my_app.c
        gcc -c -o my_app.o my_app.c
my_provider.h: my_provider.d
        dtrace -h -s my_provider.d
my_provider.o: my_app.o
        dtrace -G -s my_provider.d my_app.o

Whereas a Mac OS X Makefile might look like this.


my_app: my_app.o
        gcc -o my_app my_app.o
my_app.o: my_app.c
        gcc -c -o my_app.o my_app.c
my_provider.h: my_provider.d
        dtrace -h -s my_provider.d

Enter GNU Autoconf. Autoconf allows us to specify a template file (Makefile.in) which has placeholders for certain parts that can vary across target platforms or compile-time configuration options. The placeholders are filled in by the configure script, which checks the environment and specified options before generating a Makefile. So, in or example here: Makefile.in needs to be general enough to allow for the inclusion (or exclusion) of the ‘dtrace -G ...‘ line, as well provide the option of listing the provider object file provider.o on the final linker line. Here’s one way to meet these requirements in Makefile.in:


PROVIDER_DESC  = my_provider.d
PROVIDER_HDR   = my_provider.h
PROVIDER_OBJ   = my_provider.o
MAIN_OBJ       = my_app.o
OTHER_OBJS     = @OTHER_OBJS@
GENHDRS        = @GENHDRS@
DTFLAG         = @DTFLAG@

all: my_app

my_app: $(MAIN_OBJ) $(OTHER_OBJS)
        gcc $(DTFLAG) -o my_app $?

$(MAIN_OBJ): $(GENHDRS) my_app.c
        gcc $(DTFLAG) -c -o $@ my_app.c

$(PROVIDER_HDR): $(PROVIDER_DESC)
        dtrace -h -s $?

$(PROVIDER_OBJ): $(MAIN_OBJ)
        dtrace -G -s $(PROVIDER_DESC) $?

The variables surrounded by ‘@’ signs will be replaced by the configure script in it’s output Makefile. Depending on the needs of the target platform, these variables may contain valid values, or may be empty. For instance, a Solaris host will need ‘$OTHER_OBJS‘ to include ‘$(PROVIDER_OBJ)‘ so that the main target depends on the the provider object target (triggering it to be built). A Mac on the other hand, will need $OTHER_OBJS to be empty so that the $(PROVIDER_OBJ) target is not built.

The $(DTFLAG) variable holds a C preprocessor flag ‘-DHAVE_DTRACE’ … or not. This allows DTrace probe code to be turned on/off in the compiled program like this:


#ifdef HAVE_DTRACE MYAPP_TEST_PROBE("Hello from DTrace.n");
#endif /* HAVE_DTRACE */

The configure script is generated by autoconf (or autoreconf) from the configure.ac file. This is where we add the ‘–enable-dtrace’ flag, and the logic needed to set the substitution variables appropriately (according to the build host type). See the Autoconf reference mentioned earlier for coverage of the AC_ macros used here.


AC_INIT([my_app], [0.1])
AC_CONFIG_FILES([Makefile])
AC_CONFIG_SRCDIR([my_app.c])

dnl The GENHDRS variable will be used to determine whether to generate the
dnl provider header file (only if DTrace is enabled).
GENHDRS=""

dnl The DTFLAG variable will determine if "-DHAVE_DTRACE" is passed
dnl to the C Preprocessor at compile time (only if DTrace is enabled).
DTFLAG=""

AC_ARG_ENABLE([dtrace],
  [  --enable-dtrace     enable built-in DTrace USDT probes],
  [dtrace_requested=${enableval}],
  [dtrace_requested="no"])

if test ${dtrace_requested} = "yes"; then
  echo "*******************************"
  echo "DTrace functionality requested."
  echo "*******************************"
  AC_CHECK_HEADERS(
    [sys/sdt.h], [dtrace_supported=yes],
    [dtrace_supported=no]
  )
  if test "${dtrace_supported}" = "yes"; then
    DTFLAG="-DHAVE_DTRACE"
    GENHDRS="\$(PROVIDER_HDR)"
    dtrace_enabled=yes
    echo "***************"
    echo "DTrace engaged."
    echo "***************"
  else
    AC_MSG_ERROR([No DTrace here.])
  fi
else
  dtrace_enabled=no
fi

AC_SUBST(GENHDRS)
AC_SUBST(DTFLAG)

OTHER_OBJS=""

AC_CANONICAL_HOST
case $host in
  *solaris*)
      dnl Add the provider object to OTHER_OBJS so it will be built
      if test ${dtrace_enabled} = "yes"; then
        OTHER_OBJS="\$(PROVIDER_OBJ)"
      fi
    ;;
  *darwin*)
    dnl No extra modification needed here
    ;;
esac

AC_SUBST(OTHER_OBJS)

AC_OUTPUT

With all this in place (plus a couple helper scripts copied over from automake), we can now run ‘./configure‘ and ‘./configure --enable-dtrace‘ on both platforms – giving us correctly formatted Makefiles for our current platform.


./configure --enable-dtrace
*******************************
DTrace functionality requested.
*******************************
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /opt/local/bin/ggrep
checking for egrep... /opt/local/bin/ggrep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking sys/sdt.h usability... yes
checking sys/sdt.h presence... yes
checking for sys/sdt.h... yes
***************
DTrace engaged.
***************
checking build system type... i386-pc-solaris2.11
checking host system type... i386-pc-solaris2.11
configure: creating ./config.status
config.status: creating Makefile


mac-host$ ./configure --enable-dtrace
*******************************
DTrace functionality requested.
*******************************
checking for gcc... gcc
snip
checking sys/sdt.h usability... yes
checking sys/sdt.h presence... yes
checking for sys/sdt.h... yes
***************
DTrace engaged.
***************
checking build system type... i386-apple-darwin10.5.0
checking host system type... i386-apple-darwin10.5.0
configure: creating ./config.status
config.status: creating Makefile

mac-host$ make dtrace -h -s my_provider.d
gcc -DHAVE_DTRACE -c -o my_app.o my_app.c
gcc -DHAVE_DTRACE -o my_app my_app.o
mac-host$

dtrace_ac_example.tar.gz

arcstat.pl updated for L2ARC statistics

Three years ago, Neelakanth Nadgir released arcstat.pl: a Perl script that prints ZFS ARC statistics in a vmstat-like fashion.  I found this tool to be very useful, but it was missing support for L2ARC statistics (the L2ARC hadn’t been invented yet).  With neel’s permission, I’ve updated arcstat.pl to support L2ARC statistics.  Now it can be used to view ARC and L2ARC data side-by-side: providing a more complete view of ZFS caching performance on a system configured with L2ARC devices.

Let’s say I’m interested in watching the following stats over time:

  • ARC Accesses, Hits/sec, Misses/sec, Hit percentage
  • L2ARC Accesses, Hits/sec, Misses/sec, Hit percentage
  • ARC size and L2ARC size

I’ll specify the output fields I’m interested in as follows:


$ ./arcstat.pl -f 
>read,hits,miss,hit%,l2read,l2hits,l2miss,l2hit%,arcsz,l2size

At first, the L2ARC is cold, so we either hit in the ARC, or we miss both ARC and L2ARC (and are forced to do disk i/o):


read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
  8K    7K   272    96     272       0     272       0   912M    101M
  9K    9K   205    97     205       0     205       0   919M    102M
  7K    6K   281    96     281       0     281       0   922M    102M
  2K    2K   119    94     119       0     119       0   922M    103M
  8K    8K   517    94     517       0     517       0   930M    103M
  2K    2K   161    92     161       0     161       0   933M    103M
  1K    1K    46    97      46       0      46       0   934M    103M
  1K    1K    38    97      38       0      38       0   935M    103M
  1K    1K    21    98      21       0      21       0   911M    104M
  1K    1K    28    97      28       0      28       0   912M    104M
  1K    1K    18    98      18       0      18       0   913M    104M
  1K    1K    98    93      98       0      98       0   914M    104M

Time passes… L2ARC warms up … After a while we start to see L2ARC Hits:


read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
  3K    3K     0   100       0       0       0       0     1G      1G
  5K    5K     2    99       2       1       1      50     1G      1G
  2K    2K     0   100       0       0       0       0     1G      1G
  5K    5K     3    99       3       1       2      33     1G      1G
  4K    4K     0   100       0       0       0       0     1G      1G
  4K    4K     3    99       3       1       2      33     1G      1G
  5K    5K     3    99       3       1       2      33     1G      1G
  4K    4K     2    99       2       1       1      50     1G      1G
  5K    5K    17    99      17       4      13      23     1G      1G

More time passes… As the ARC churns, it’s hit ratio drops because it can’t cover the entire working set.  L2ARC accesses increase, as ARC misses are picked up by the L2ARC (becoming L2ARC hits):

read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
  8K    8K    13    99      13      12       1      92     1G      3G
  7K    7K    18    99      18      12       6      66     1G      3G
  8K    8K    83    98      83      79       4      95     1G      3G
 768   182   586    23     586     532      54      90     1G      3G
  3K    2K   521    84     521     510      11      97     1G      3G
  8K    7K   620    92     620     619       1      99     1G      3G
  9K    9K   662    93     662     660       2      99     1G      3G
  4K    3K   732    82     732     732       0     100     1G      3G
  5K    5K   345    93     345      76     269      22     1G      3G
 13K   13K   496    96     496     372     124      75     1G      3G
  2K    1K   742    68     742     704      38      94     1G      3G
  2K    1K   780    62     780     757      23      97     1G      3G
  2K    1K   781    60     781     781       0     100     1G      3G
  2K    1K   714    70     714     709       5      99     1G      3G
  2K    1K   846    61     846     831      15      98     1G      3G
  2K    2K   781    73     781     767      14      98     1G      3G
  4K    3K   649    86     649     590      59      90     1G      3G

I’ve found this tool (used along with ‘iostat’ or ‘zpool iostat -v’) to be a nice way to get a feel for how the ARC + L2ARC are performing. It will definitely give you a feel for how long the L2ARC takes to warm-up for your workload. The above examples were snipped from a timespan of 2 hours while compiling the illumos gate.