Parent

Raindrops::Aggregate::PMQ

Aggregate + POSIX message queues support for Ruby 1.9 and Linux

This class is duck-type compatible with Aggregate and allows us to aggregate and share statistics from multiple processes/threads aided POSIX message queues. This is designed to be used with the Raindrops::LastDataRecv Rack application, but can be used independently on compatible Runtimes.

Unlike the core of raindrops, this is only supported on Ruby 1.9 and Linux 2.6. Using this class requires the following additional RubyGems or libraries:

Design

There is one master thread which aggregates statistics. Individual worker processes or threads will write to a shared POSIX message queue (default: “/raindrops”) that the master reads from. At a predefined interval, the master thread will write out to a shared, anonymous temporary file that workers may read from

Setting :worker_interval and :master_interval to 1 will result in perfect accuracy but at the cost of a high synchronization overhead. Larger intervals mean less frequent messaging for higher performance but lower accuracy.

Attributes

nr_dropped[R]

returns the number of dropped messages sent to a POSIX message queue if non-blocking operation was desired with :lossy

Public Class Methods

new(params = {}) click to toggle source

Creates a new Raindrops::Aggregate::PMQ object

  Raindrops::Aggregate::PMQ.new(options = {})  -> aggregate

options is a hash that accepts the following keys:

  • :queue - name of the POSIX message queue (default: “/raindrops”)

  • :worker_interval - interval to send to the master (default: 10)

  • :master_interval - interval to for the master to write out (default: 5)

  • :lossy - workers drop packets if master cannot keep up (default: false)

  • :aggregate - Aggregate object (default: Aggregate.new)

  • :mq_umask - umask for creatingthe POSIX message queue (default: 0666)

    # File lib/raindrops/aggregate/pmq.rb, line 65
65:   def initialize(params = {})
66:     opts = {
67:       :queue => ENV["RAINDROPS_MQUEUE"] || "/raindrops",
68:       :worker_interval => 10,
69:       :master_interval => 5,
70:       :lossy => false,
71:       :mq_attr => nil,
72:       :mq_umask => 0666,
73:       :aggregate => Aggregate.new,
74:     }.merge! params
75:     @master_interval = opts[:master_interval]
76:     @worker_interval = opts[:worker_interval]
77:     @aggregate = opts[:aggregate]
78:     @worker_queue = @worker_interval ? [] : nil
79:     @mutex = Mutex.new
80: 
81:     @mq_name = opts[:queue]
82:     mq = POSIX_MQ.new @mq_name, :w, opts[:mq_umask], opts[:mq_attr]
83:     Tempfile.open("raindrops_pmq") do |t|
84:       @wr = File.open(t.path, "wb")
85:       @rd = File.open(t.path, "rb")
86:     end
87:     @cached_aggregate = @aggregate
88:     flush_master
89:     @mq_send = if opts[:lossy]
90:       @nr_dropped = 0
91:       mq.nonblock = true
92:       mq.method :trysend
93:     else
94:       mq.method :send
95:     end
96:   end

Public Instance Methods

<<(val) click to toggle source

adds a sample to the underlying Aggregate object

     # File lib/raindrops/aggregate/pmq.rb, line 99
 99:   def << val
100:     if q = @worker_queue
101:       q << val
102:       if q.size >= @worker_interval
103:         mq_send(q) or @nr_dropped += 1
104:         q.clear
105:       end
106:     else
107:       mq_send(val) or @nr_dropped += 1
108:     end
109:   end
aggregate() click to toggle source

Loads the last shared Aggregate from the master thread/process

     # File lib/raindrops/aggregate/pmq.rb, line 150
150:   def aggregate
151:     @cached_aggregate ||= begin
152:       flush
153:       Marshal.load(synchronize(@rd, RDLOCK) do |rd|
154:         IO.pread rd.fileno, rd.stat.size, 0
155:       end)
156:     end
157:   end
count() click to toggle source

proxy for Aggregate#count

     # File lib/raindrops/aggregate/pmq.rb, line 208
208:   def count; aggregate.count; end
each() click to toggle source

proxy for Aggregate#each

     # File lib/raindrops/aggregate/pmq.rb, line 235
235:   def each; aggregate.each { |*args| yield(*args) }; end
each_nonzero() click to toggle source

proxy for Aggregate#each_nonzero

     # File lib/raindrops/aggregate/pmq.rb, line 238
238:   def each_nonzero; aggregate.each_nonzero { |*args| yield(*args) }; end
flush() click to toggle source

flushes the local queue of the worker process, sending all pending data to the master. There is no need to call this explicitly as :worker_interval defines how frequently your queue will be flushed

     # File lib/raindrops/aggregate/pmq.rb, line 199
199:   def flush
200:     if q = @local_queue && ! q.empty?
201:       mq_send q
202:       q.clear
203:     end
204:     nil
205:   end
flush_master() click to toggle source

Flushes the currently aggregate statistics to a temporary file. There is no need to call this explicitly as :worker_interval defines how frequently your data will be flushed for workers to read.

     # File lib/raindrops/aggregate/pmq.rb, line 162
162:   def flush_master
163:     dump = Marshal.dump @aggregate
164:     synchronize(@wr, WRLOCK) do |wr|
165:       wr.truncate 0
166:       IO.pwrite wr.fileno, dump, 0
167:     end
168:   end
master_loop() click to toggle source

Starts running a master loop, usually in a dedicated thread or process:

  Thread.new { agg.master_loop }

Any worker can call agg.stop_master_loop to stop the master loop (possibly causing the thread or process to exit)

     # File lib/raindrops/aggregate/pmq.rb, line 123
123:   def master_loop
124:     buf = ""
125:     a = @aggregate
126:     nr = 0
127:     mq = POSIX_MQ.new @mq_name, :r # this one is always blocking
128:     begin
129:       if (nr -= 1) < 0
130:         nr = @master_interval
131:         flush_master
132:       end
133:       mq.shift(buf)
134:       data = begin
135:         Marshal.load(buf) or return
136:       rescue ArgumentError, TypeError
137:         next
138:       end
139:       Array === data ? data.each { |x| a << x } : a << data
140:     rescue Errno::EINTR
141:     rescue => e
142:       warn "Unhandled exception in #{__FILE__}:#{__LINE__}: #{e}"
143:       break
144:     end while true
145:     ensure
146:       flush_master
147:   end
max() click to toggle source

proxy for Aggregate#max

     # File lib/raindrops/aggregate/pmq.rb, line 211
211:   def max; aggregate.max; end
mean() click to toggle source

proxy for Aggregate#mean

     # File lib/raindrops/aggregate/pmq.rb, line 220
220:   def mean; aggregate.mean; end
min() click to toggle source

proxy for Aggregate#min

     # File lib/raindrops/aggregate/pmq.rb, line 214
214:   def min; aggregate.min; end
outliers_high() click to toggle source

proxy for Aggregate#outliers_high

     # File lib/raindrops/aggregate/pmq.rb, line 229
229:   def outliers_high; aggregate.outliers_high; end
outliers_low() click to toggle source

proxy for Aggregate#outliers_low

     # File lib/raindrops/aggregate/pmq.rb, line 226
226:   def outliers_low; aggregate.outliers_low; end
std_dev() click to toggle source

proxy for Aggregate#std_dev

     # File lib/raindrops/aggregate/pmq.rb, line 223
223:   def std_dev; aggregate.std_dev; end
stop_master_loop() click to toggle source

stops the currently running master loop, may be called from any worker thread or process

     # File lib/raindrops/aggregate/pmq.rb, line 172
172:   def stop_master_loop
173:     sleep 0.1 until mq_send(false)
174:     rescue Errno::EINTR
175:       retry
176:   end
sum() click to toggle source

proxy for Aggregate#sum

     # File lib/raindrops/aggregate/pmq.rb, line 217
217:   def sum; aggregate.sum; end
to_s(*args) click to toggle source

proxy for Aggregate#to_s

     # File lib/raindrops/aggregate/pmq.rb, line 232
232:   def to_s(*args); aggregate.to_s(*args); end

Disabled; run with --debug to generate this.

[Validate]

Generated with the Darkfish Rdoc Generator 1.1.6.