Object
Aggregate + POSIX message queues support for Ruby 1.9 and Linux
This class is duck-type compatible with Aggregate and allows us to aggregate and share statistics from multiple processes/threads aided POSIX message queues. This is designed to be used with the Raindrops::LastDataRecv Rack application, but can be used independently on compatible Runtimes.
Unlike the core of raindrops, this is only supported on Ruby 1.9 and Linux 2.6. Using this class requires the following additional RubyGems or libraries:
aggregate (tested with 0.2.2)
io-extra (tested with 1.2.3)
posix_mq (tested with 1.0.0)
There is one master thread which aggregates statistics. Individual worker processes or threads will write to a shared POSIX message queue (default: “/raindrops”) that the master reads from. At a predefined interval, the master thread will write out to a shared, anonymous temporary file that workers may read from
Setting :worker_interval and :master_interval to 1 will result in perfect accuracy but at the cost of a high synchronization overhead. Larger intervals mean less frequent messaging for higher performance but lower accuracy.
Creates a new Raindrops::Aggregate::PMQ object
Raindrops::Aggregate::PMQ.new(options = {}) -> aggregate
options is a hash that accepts the following keys:
:queue - name of the POSIX message queue (default: “/raindrops”)
:worker_interval - interval to send to the master (default: 10)
:master_interval - interval to for the master to write out (default: 5)
:lossy - workers drop packets if master cannot keep up (default: false)
:aggregate - Aggregate object (default: Aggregate.new)
:mq_umask - umask for creatingthe POSIX message queue (default: 0666)
# File lib/raindrops/aggregate/pmq.rb, line 65 65: def initialize(params = {}) 66: opts = { 67: :queue => ENV["RAINDROPS_MQUEUE"] || "/raindrops", 68: :worker_interval => 10, 69: :master_interval => 5, 70: :lossy => false, 71: :mq_attr => nil, 72: :mq_umask => 0666, 73: :aggregate => Aggregate.new, 74: }.merge! params 75: @master_interval = opts[:master_interval] 76: @worker_interval = opts[:worker_interval] 77: @aggregate = opts[:aggregate] 78: @worker_queue = @worker_interval ? [] : nil 79: @mutex = Mutex.new 80: 81: @mq_name = opts[:queue] 82: mq = POSIX_MQ.new @mq_name, :w, opts[:mq_umask], opts[:mq_attr] 83: Tempfile.open("raindrops_pmq") do |t| 84: @wr = File.open(t.path, "wb") 85: @rd = File.open(t.path, "rb") 86: end 87: @cached_aggregate = @aggregate 88: flush_master 89: @mq_send = if opts[:lossy] 90: @nr_dropped = 0 91: mq.nonblock = true 92: mq.method :trysend 93: else 94: mq.method :send 95: end 96: end
adds a sample to the underlying Aggregate object
# File lib/raindrops/aggregate/pmq.rb, line 99 99: def << val 100: if q = @worker_queue 101: q << val 102: if q.size >= @worker_interval 103: mq_send(q) or @nr_dropped += 1 104: q.clear 105: end 106: else 107: mq_send(val) or @nr_dropped += 1 108: end 109: end
Loads the last shared Aggregate from the master thread/process
# File lib/raindrops/aggregate/pmq.rb, line 150 150: def aggregate 151: @cached_aggregate ||= begin 152: flush 153: Marshal.load(synchronize(@rd, RDLOCK) do |rd| 154: IO.pread rd.fileno, rd.stat.size, 0 155: end) 156: end 157: end
proxy for Aggregate#count
# File lib/raindrops/aggregate/pmq.rb, line 208 208: def count; aggregate.count; end
proxy for Aggregate#each
# File lib/raindrops/aggregate/pmq.rb, line 235 235: def each; aggregate.each { |*args| yield(*args) }; end
proxy for Aggregate#each_nonzero
# File lib/raindrops/aggregate/pmq.rb, line 238 238: def each_nonzero; aggregate.each_nonzero { |*args| yield(*args) }; end
flushes the local queue of the worker process, sending all pending data to the master. There is no need to call this explicitly as :worker_interval defines how frequently your queue will be flushed
# File lib/raindrops/aggregate/pmq.rb, line 199 199: def flush 200: if q = @local_queue && ! q.empty? 201: mq_send q 202: q.clear 203: end 204: nil 205: end
Flushes the currently aggregate statistics to a temporary file. There is no need to call this explicitly as :worker_interval defines how frequently your data will be flushed for workers to read.
# File lib/raindrops/aggregate/pmq.rb, line 162 162: def flush_master 163: dump = Marshal.dump @aggregate 164: synchronize(@wr, WRLOCK) do |wr| 165: wr.truncate 0 166: IO.pwrite wr.fileno, dump, 0 167: end 168: end
Starts running a master loop, usually in a dedicated thread or process:
Thread.new { agg.master_loop }
Any worker can call agg.stop_master_loop to stop the master loop (possibly causing the thread or process to exit)
# File lib/raindrops/aggregate/pmq.rb, line 123 123: def master_loop 124: buf = "" 125: a = @aggregate 126: nr = 0 127: mq = POSIX_MQ.new @mq_name, :r # this one is always blocking 128: begin 129: if (nr -= 1) < 0 130: nr = @master_interval 131: flush_master 132: end 133: mq.shift(buf) 134: data = begin 135: Marshal.load(buf) or return 136: rescue ArgumentError, TypeError 137: next 138: end 139: Array === data ? data.each { |x| a << x } : a << data 140: rescue Errno::EINTR 141: rescue => e 142: warn "Unhandled exception in #{__FILE__}:#{__LINE__}: #{e}" 143: break 144: end while true 145: ensure 146: flush_master 147: end
proxy for Aggregate#max
# File lib/raindrops/aggregate/pmq.rb, line 211 211: def max; aggregate.max; end
proxy for Aggregate#mean
# File lib/raindrops/aggregate/pmq.rb, line 220 220: def mean; aggregate.mean; end
proxy for Aggregate#min
# File lib/raindrops/aggregate/pmq.rb, line 214 214: def min; aggregate.min; end
proxy for Aggregate#outliers_high
# File lib/raindrops/aggregate/pmq.rb, line 229 229: def outliers_high; aggregate.outliers_high; end
proxy for Aggregate#outliers_low
# File lib/raindrops/aggregate/pmq.rb, line 226 226: def outliers_low; aggregate.outliers_low; end
proxy for Aggregate#std_dev
# File lib/raindrops/aggregate/pmq.rb, line 223 223: def std_dev; aggregate.std_dev; end
stops the currently running master loop, may be called from any worker thread or process
# File lib/raindrops/aggregate/pmq.rb, line 172 172: def stop_master_loop 173: sleep 0.1 until mq_send(false) 174: rescue Errno::EINTR 175: retry 176: end
Disabled; run with --debug to generate this.
Generated with the Darkfish Rdoc Generator 1.1.6.