Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

Dinakar Guniguntala
On Sat, Apr 23, 2005 at 03:30:59PM -0700, Paul Jackson wrote:
> The top cpuset holds the kernel threads that are pinned to a particular
> cpu or node.  It's not right that their cpusets cpus_allowed is empty,
> which is what I guess the "0" in the cpus_allowed column above means.
> (Even if the "0" means CPU 0, that still conflicts with kernel threads
> on CPUs 1-7.)

Yes, I meant cpus_allowed is empty

>
> We might get away with it on cpus, because we don't change the tasks
> cpus_allowed to match the cpusets cpus_allowed (we don't call
> set_cpus_allowed, from kernel/cpuset.c) _except_ when someone rebinds
> that task to its cpuset by writing its pid into the cpuset tasks file.
> So for as long as no one tries to rebind the per-cpu or per-node
> kernel threads, no one will notice that they in a cpuset with an
> empty cpus_allowed.

True.

>  4) There are some tasks that _do_ require to run on the same cpus as
>     the tasks you would assign to isolated cpusets.  These kernel threads,
>     such as for example the migration and ksoftirqd threads, must be setup
>     well before user code is run that can configure job specific isolated
>     cpusets, so these tasks need a cpuset to run in that can be created
>     during the system boot, before init (pid == 1) starts up.  This cpuset
>     is the top cpuset.

And those processes (kernel threads) will continue to run on their cpus

>
> I don't understand why what's there now isn't sufficient.  I don't see
> that this patch provides any capability that you can't get just by
> properly placing tasks in cpusets that have the desired cpus and nodes.
> This patch leaves the per-cpu kernel threads with no cpuset that allows
> what they need, and it complicates the semantics of things, in ways that
> I still don't entirely understand.

You are forgetting the fact that the scheduler is still load balancing
across all CPUs and tries to pull tasks only to find that the task's
cpus_allowed mask prevents it from being moved

>
> You don't need to isolate a set of cpus; you need to isolate a set of
> processes.  So long as you can create non-overlapping cpusets, and
> assign processes to them, I don't see where it matters that you cannot
> prohibit the creation of overlapping cpusets, or in the case of the top
> cpuset, why it matters that you cannot _disallow_ allowed cpus
> or memory nodes in existing cpusets.
>

I am working on a minimalistic design right now and will get back in
a day or two

        -Dinakar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

Paul Jackson
Dinakar, replying to pj:

> > I don't understand why what's there now isn't sufficient.  I don't see
> > that this patch provides any capability that you can't get just by
> > properly placing tasks in cpusets that have the desired cpus and nodes.
> > This patch leaves the per-cpu kernel threads with no cpuset that allows
> > what they need, and it complicates the semantics of things, in ways that
> > I still don't entirely understand.
>
> You are forgetting the fact that the scheduler is still load balancing
> across all CPUs and tries to pull tasks only to find that the task's
> cpus_allowed mask prevents it from being moved

Well, I haven't forgotten, but I am having a difficult time figuring out
what your real (most likely just one or two) essential requirements are.

A few days ago, you provided a six step list, under the introduction:
> Ok, Let me begin at the beginning and attempt to define what I am
> doing here

I suspect those six steps were not really your essential requirements,
but one possible procedure that accomplishes them.

So far I am guessing that your requirement(s) are one or both of the
following two items:

 (1) avoid having the scheduler wasting too much time trying to
     load balance tasks that only turn out to be not allowed on
     the cpus the scheduler is considering, and/or
 (2) provide improved administrative control of a system by being
     able to construct a cpuset that is guaranteed to have no
     overlap of allowed cpus with its parent or any other cpuset
     not descendent from it.

If (1) is one of your essential requirements, then I have described a
couple of implementations that mark some existing cpusets to form a
partition (in the mathematical sense of a disjoint covering of subsets)
of the system to define isolated scheduler domains.  I did this without
adding any additional bitmasks to each cpuset.

If (2) is one of your essential requirements, then I believe this can be
done with the current cpuset kernel code, entirely with additional user
level code.

> I am working on a minimalistic design right now and will get back in
> a day or two

Good.

Hopefully, you will also be able to get through my thick head what you
essential requirement(s) is or are.

--
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[hidden email]> 1.650.933.1373, 1.925.600.0401
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/