* More locking issues

Adding someone to a queue or list, requires allocating some small amount of
memory (a data struct) to describe the entity that's going to be waiting.
Let's say we use kmalloc().  There's two issues:

1. kmalloc() could block, put the thread to WAIT, until enough memory is
freed.  Recall kmalloc takes a 2nd arg w/ flags to control its behavior.

So if add_to(wait queue) could block, then we can't use a "quick" spinlock!
We then will have to change our internal struct rwsem lock from spinlock to
a mutex.  Otherwise, you'd have to use kmalloc w/ flags that it won't block,
but be prepared for kmalloc to fail (return NULL), and handle it somehow.

2. calling add_to(wait queue) consumes some memory (even a small amount).

Problem: the current implementation allows an unlimited number of readers to
be waiting while a writer is running (in CS) or waiting.  We should never
allow unlimited growth of resources inside the kernel!  Most implementations
will have a cap to the maximum amount of resources consumed.  A useful cap
for "queue waiters" is to have a max length of the waiting queues.

So we'll add an "int max_owners" to struct rwsem.  Let's say it's configured
by default to, 10 (often a good idea to allow admins to set this max within
some reasonable bounds, say 2 to 50).  The reason to limit even how much the
max can be configured to, is to prevent admins from [mis]setting it to some
huge number (that'll be equivalent to infinity).

You may also want to limit how many active readers can enter their CS: by
limiting this number, you ensure that a writer waiting on existing readers,
won't have to wait "too long".  If you wanted to do this, you'd need to add
a max_active_readers and manage that as well under the lock 'L'.

Recall that a rwsem is useful when you have many readers but only one
writer.  If you find that too many writers want to get into their CS, then
you may want to limit the size of the waiting_writers queue as well (a good
idea anyway not to run out of memory).  You may also consider that if no. of
waiting writers is >N (for small N), maybe printk("consider using a mutex
not a rwsem").

Q: What if readers are waiting but a writer is taking "too long"? This is
the same as having any one in a waitq, waiting for something else to happen.

A: It's never a good idea for anyone to sit in a CS for too long.  What is
too long depends on the particular situation.  The longer one is blocked
from entering a CS b/c someone else is taking too long in their CS, the
worse the system throughput will be -- a form of starvation.

* A generalized producer consumer asynchronous queuing system

A generalization of rwsem.

Producer: a entity who asks for some work to be done.
Consumer: another entity who actually does the work, some time later.

Often useful when the effort to "produce" (ask) for work is much smaller
than the effort to "consume" (actually do) the work.  Why?  If the producer
has to wait a long time for the actual work, then they just sit WAITing,
doing nothing else useful; instead, the producer can "submit" the job to
someone else to do, and ask to be told/informed when the work is done.  That
way, the producer can go and do other things while they wait for the
consumers to perform the work itself.

Examples: kernel logs, I/O subsystems (e.g., getting a block from disk).

If you issue a read(2) syscall, you are blocked from doing anything else.
Sometimes we say that 'reads' are synchronous, b/c you have to wait for
them.

But if you call aio_read(2) that call returns immediately even if the data
is not available yet.  You have to pass a callback fxn to aio_read, and the
kernel will invoke your callback when the data actually arrived (which could
be many milliseconds and even seconds later).


A producer-consumer async system has:

1. one or more producers (e.g., users or applications submitting work to be
done).

2. One of more consumer threads that process the submitted jobs in some
order.

3. A queue of jobs submitted, in some order.  You can use one FCFS queue, or
have multiple queues, with priorities.

4. A limit to the size of the queue, so it won't grow unbounded.

5. Internal locks (spinlocks? mutexes? others) that protect internal data
structures.

Normal operation:

1. Users submit a job, which gets queued for later processing, then users
return immediately, and they can go do something else.  If producers want to
be informed when the work is done, they have to provide a callback function
(and possibly a "void*arg" for the consumer to place data into).

Note: the telltale sign of an async processing system, is when the function
you call has some sort of a callback fxn and possibly a void*.

2. Consumers pick up jobs, one at a time, and process them.  Recall
processing the jobs takes longer than it took to produce them.  When
consumers are done, then return results back to the producers.

Q: Since producing work is faster that consuming it, how do we prevent too
many producers adding too much work to the queue?

A: We have to limit the size of the queue.  Jobs can be added to a queue up
to some MAX, and we can return immediately to the producer.  But if we reach
the MAX queue length, then we have to block new producers from adding more
work to the queue.  Same as with rwsem code: we want to put them into a WAIT
queue (producers waiting to add work to the queue itself).  That's better
than returning some error or "try again later" b/c users will try in an
inf. loop.

In a producer-consumer system, blocking new producers from doing any work,
and actually putting them to sleep (WAIT state), is called "throttling the
heavy writers."  Here, the "writers" are those who want to add work to the
queue.